1. Dynamic Connectivity pb

pb statement

a set of N obj, indexed by 0,1,...,N-1 ⇒ * UNION: connect objects void union(int p, int q) * FIND: is there a path connecting 2 obj? boolean connected(int p, int q)

ex:

connect components(联通分支): max set of obj …

STATISTICS, BUSINESS QUESTIONS, AND LEARNING TECHNIQUES

2 different kinds of statistics:

• descriptive statistics

ex. median — describes data, but cannot generalize beyong that

• inferential statistics

ex. t-testing — inferences beyond the data techniques leveraged for machine learning and prediction

supervised learning (clf, reg), unsupervised learning (clustering, dim-reduction) → UL often used in a …

DATA CLEANING

ex. deal with missing data, entity resolution, unit mismatch, ...

deal with non-ideal samples ⇒ tradeoff between simplicity and accuracy.

DATA QUALITY PROBLEMS

data quality problems:

• Conversions in complex pipelines can mess up data
• Combining multiple datasets can result in errrors
• Data degrades in accuracy or loses value over time …

RELATIONAL DATABASE

review: key data management concepts:

• data model
• schema
• relational data model

structured data: have a specific schema to start with

relationl database: a set of relations. 2 parts to a Relation:

• schema: name of relation, name and type of columns

• instance:

any data at given time (cardinality:=nb …

KEY DATA MANAGEMENT CONCEPTS

data model: collection of concepts for describing data schema: a description of a particular collection of data using a given data model

structure spectrum:
semi-structured data: apply schema after creating data.

FILES

files: named collection of bytes, in hierarchical namespace (but: In a Content-Addressable Storage system …

PYTHON SPARK (PYSPARK)

a spark prog has 2 programs:

• dirver program: runs on driver machine
• worker program: runs on local threads or cluster nodes

a spark prog first creates a SparkContext object:

• tells how and where to access a cluster
• shell will automatically create the sc varible
• in iPython: use …

Scrapy是用来爬取数据的很流行的包, 这里小记一下. 以前几天做的一个爬虫为例子, 这个爬虫把韩寒一个app的前九百多期的文章抓了下来.

I. installation

scrapy的安装参考: http://scrapy-chs.readthedocs.org/zh_CN/latest/topics/ubuntu.html

(直接pip安装的好像缺少什么包)

II. prerequisite

XPath

需要 …

INF580(programmation par contraintes) 大概是在X学到的最有用的一门课, 它让我能够用把运筹学(MAP557)里学到的东西和计算机结合起来: 用电脑的力量解决(大规模)运筹问题.

这 …

压缩pdf

用convert 只简单指定resize好像不太好使:

convert -resize 50% input.pdf out.pdf

用gs(http://blog.sciencenet.cn/blog-467089-773990.html):

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -dNOPAUSE -dQUIET -dBATCH -sOutputFile=out.pdf input.pdf

后来发现老是compress以后的结果不好, 看 …

在处理大量数的时候, 如果输出类似 "process i out of n files..." 这样的内容来指示进度的话, 虽然可以显示目前的进度(用来安慰等待 …