2. Authors
  3. mx

[Spark MOOC note] Lec6. Structured Data

Category notes spark Series Part 3 of «spark MOOC»

RELATIONAL DATABASE

review: key data management concepts:

  • data model
  • schema
  • relational data model

structured data: have a specific schema to start with

relationl database: a set of relations. 2 parts to a Relation:

  • schema: name of relation, name and type of columns

  • instance:

any data at given time (cardinality:=nb …

[Spark MOOC note] Lec5. Semi-structured Data

Category notes spark Series Part 2 of «spark MOOC»

KEY DATA MANAGEMENT CONCEPTS

data model: collection of concepts for describing data schema: a description of a particular collection of data using a given data model

structure spectrum:
semi-structured data: apply schema after creating data.

FILES

files: named collection of bytes, in hierarchical namespace (but: In a Content-Addressable Storage system …

[Spark MOOC note] Lec4. Spark Essentials

Category notes spark Series Part 1 of «spark MOOC»

PYTHON SPARK (PYSPARK)

a spark prog has 2 programs:

  • dirver program: runs on driver machine
  • worker program: runs on local threads or cluster nodes

a spark prog first creates a SparkContext object:

  • tells how and where to access a cluster
  • shell will automatically create the sc varible
  • in iPython: use …

一个简单的python进度条

Category tech python

在处理大量数的时候, 如果输出类似 "process i out of n files..." 这样的内容来指示进度的话, 虽然可以显示目前的进度(用来安慰等待 …

putty使用备忘

Category soft ssh

最近要用SSH连接服务器, Windows下面当然就是用putty了, 遇到的问题总结一下.

保存session

打开putty.exe以后, 输入服务器ip, 之后先别点击登录, 先保存一下session下一次就不用再输入了:

之后点击登录就好 …