Character

Posted on 2018-05-13

Common used functions:

isLetter()
isDigit()
isWhitespace()
isUpperCase()
isLowerCase()
toUpperCase()
toLowerCase()
toString()

Java-Number & Math package

Posted on 2018-05-13 | Edited on 2018-05-30

Common used functions:

import Math

compareTo()
equals()
valueOf()
toString()
parseInt()
abs()
ceil(): return minimum greater or equal to target number
floor(): return maximum less or equal to target number
min():
max():
exp()
log()
pow()
sqrt()
sin()
cos()
tan()
asin()
acos()
atan()
random()

Docker

Posted on 2018-05-13 | Edited on 2018-06-08

Docker is an open source container engine, which is written by Golang.

It allows developers to pack app into a lightweight, portable container and deploy on any Linux machine to achieve virtualization. It adopts sandbox mechanism, which has no interface between containers, so that container costs less.

Architecture

Docker uses Cli-Server mode to manage and build container by remote APIs.

1
2
3

Docker        Object-oriented
Container     Object
Image         Class

Docker Images: The template to build Docker Container.
Docker Container: One or more apps run in container independently.
Docker Client: keep in touch with daemon by command lines and Docker APIs.
Docker Host: a physical or virtual machine to execute Docker daemon and containers.
Docker Registry: a Docker warehouse to store images.
Docker Machine: a command line tool to install Docker in target platform, like VirtualBox, Digital Ocean.

Implements

Web app, Back-end app auto-pack and deployment

Commands

Install docker:
install docker
1
2
3
```
2. Search commands:
```docker

You will see all commands.

Dockerfile

docker build install all setting environments.
commands:
FROM; RUN; ADD; COPY; CMD; EXPOSE; WORKDIR;
MAINTAINER; ENV; ENTRYPOINT; USER; VOLUME

FROM ubuntu
MAINTAINER xinxin tang
RUN sed -i 's/archive.ubuntu.com/mirrrors.ustc.edu.cn/g' /etc/apt/sources.list
RUN apt-get update
RUN apt-get install -y nginx
COPY index.html /var/www/html
ENTRYPOINT ["/usr/sbin/nginx", "daemon off;"]
EXPOSE 80

Set commands in dockerfile;

Storage - Volume

1 2	docker run -v /usr/xinxin/html nginx -v: persist storage

Multi-App - docker-compose

set all configurations for packing applications together.
dockerfile;
yaml

commands:
build; command; depends_on; ports; volumes; image;
up; stop; rm; logs; ps

json

Posted on 2018-05-13

JSON: JavaScript Object Notation

Lightweight text data exchange format.

Why JSON?
More concise;
No end tags;
Faster read/write ability than XML

Syntax
Data stores in Key/Value pair;
Data is splitted by comma;
Brace stores object;
Bracket stores array;

Values
Number:

1	{ "age":30 }

String

Boolean:

1	{ "flag":true }

Array:

{
  "sites": [
    { "name":"菜鸟教程" , "url":"www.runoob.com" }, 
    { "name":"google" , "url":"www.google.com" }, 
    { "name":"微博" , "url":"www.weibo.com" }
  ]
}

Object:

1	{ "name":"Tutorials" , "url":"www.google.com" }

null:

{ "runoob":null }

web_develop

Posted on 2018-05-12 | Edited on 2018-05-13

HTML is HyperText Markup Language.
CSS is a language that describes the style of an HTML document.
JavaScript is a programming language of HTML and the Web.

Why study them?

HTML to define the content of web pages;
CSS to specify the layout of web pages;
JavaScript to program the behavior of web pages.

JavaScript Libraries
JQuery
Angular
React: a declarative, efficient, and flexible JavaScript library for building user interfaces.

Node.js
it is an open source server environment, which allows us to run JavaScript on the server.

Django
it is a high-level Python web framework that makes us easier to build better web apps quickly with less code.

html5

Posted on 2018-05-12 | Edited on 2018-05-30

HyperText Markup Language is short as HTML.

The <!DOCTYPE html> declaration defines this document to be HTML5
The element is the root element of an HTML page
The element contains meta information about the document
The element specifies a title for the document The <body> element contains the visible page content The <h1> element defines a large heading The element defines a paragraph <h2 id="HTML-Page-Structure"><a href="#HTML-Page-Structure" class="headerlink" title="HTML Page Structure"></a>HTML Page Structure</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre>1 2 3 4 5 6 7 8 9 10 11 </pre></td><td class="code"><pre><!DOCTYPE html> <html> <head> <title>Page Title</title> </head> <body> <h1>My First Heading</h1> My first paragraph. </body> </html> </pre></td></tr></table></figure> Link: <figure class="highlight plain"><table><tr><td class="gutter"><pre>1 </pre></td><td class="code"><pre><a href="https://www.google.com">This is a link</a> </pre></td></tr></table></figure> Images: <figure class="highlight plain"><table><tr><td class="gutter"><pre>1 </pre></td><td class="code"><pre><img src="w3schools.jpg" alt="W3Schools.com" width="104" height="142"> </pre></td></tr></table></figure> Button: <figure class="highlight plain"><table><tr><td class="gutter"><pre>1 </pre></td><td class="code"><pre><button>Click me</button> </pre></td></tr></table></figure> List: <figure class="highlight plain"><table><tr><td class="gutter"><pre>1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 </pre></td><td class="code"><pre><h2>An Unordered HTML List</h2> <ul> <li>Coffee</li> <li>Tea</li> <li>Milk</li> </ul> <h2>An Ordered HTML List</h2> <ol> <li>Coffee</li> <li>Tea</li> <li>Milk</li> </ol> </pre></td></tr></table></figure> Heading: <h1> Most important heading, <h6> the least important heading.</h6></h1></h1></body>

Bigger Headings using CSS property:

1	<h1 style="font-size:60px;">Heading 1</h1>

Horizontal Rules:

defines a thematic break in a HTML page.

<h1>This is heading 1</h1>
<p>This is some text.</p>
<hr>
<h2>This is heading 2</h2>
<p>This is some other text.</p>
<hr>

Element:
it is a container for metadata. HTML metadata is data about the HTML document. Metadata is not displayed.

Paragraphs

1 2	<p>This is a paragraph.</p> <p>This is another paragraph.</p>

<p>
This paragraph
contains a lot of lines
in the source code,
but the browser
ignores it.
</p>

<p>
This paragraph
contains      a lot of spaces
in the source     code,
but the    browser
ignores it.
</p>

<p>
The number of lines in a paragraph depends on the size of the browser window. If you resize the browser window, the number of lines in this paragraph will change.
</p>

Line Breaks

1	<p>This is<br>a paragraph<br>with line breaks.</p>

 element is displayed in a fixed-width font.
1
2
3
4
5
6
7
8
9
<pre>
  My Bonnie lies over the ocean.

  My Bonnie lies over the sea.

  My Bonnie lies over the ocean.

  Oh, bring back my Bonnie to me.
</pre>

Styles

Setting the style using CSS.

1	<tagname style="property:value;">

1.Background Color

<body style="background-color:powderblue;">

  <h1>This is a heading</h1>
  <p>This is a paragraph.</p>

</body>

2.Text Color

1 2	<h1 style="color:blue;">This is a heading</h1> <p style="color:red;">This is a paragraph.</p>

3.Fonts

1 2	<h1 style="font-family:verdana;">This is a heading</h1> <p style="font-family:courier;">This is a paragraph.</p>

4.Text Size

1 2	<h1 style="font-size:300%;">This is a heading</h1> <p style="font-size:160%;">This is a paragraph.</p>

5.Text Alignment

1 2	<h1 style="text-align:center;">Centered Heading</h1> <p style="text-align:center;">Centered paragraph.</p>

Formatting elements

- Bold text
- Important text
- Italic text
- Emphasized text
- Marked text
- Small text
- Deleted text
- Inserted text
_{- Subscript text
^{- Superscript text}}

1	<b>This text is bold</b>

1	<strong>This text is strong</strong>

Quotation and Citation Elements

Defines an abbreviation or acronym

Defines contact information for the author/owner of a document
Defines the text direction

Defines a section that is quoted from another source
Defines the title of a work
Defines a short inline quotation

Comment tags

1	<!-- Write your comments here -->

Color

Background color

1 2	<h1 style="background-color:DodgerBlue;">Hello World</h1> <p style="background-color:Tomato;">Lorem ipsum...</p>

Text color

1
2
3

<h1 style="color:Tomato;">Hello World</h1>
<p style="color:DodgerBlue;">Lorem ipsum...</p>
<p style="color:MediumSeaGreen;">Ut wisi enim...</p>

3.Border color

1
2
3

<h1 style="border:2px solid Tomato;">Hello World</h1>
<h1 style="border:2px solid DodgerBlue;">Hello World</h1>
<h1 style="border:2px solid Violet;">Hello World</h1>

4.Color values

<h1 style="background-color:rgb(255, 99, 71);">...</h1>
<h1 style="background-color:#ff6347;">...</h1>
<h1 style="background-color:hsl(9, 100%, 64%);">...</h1>
<h1 style="background-color:rgba(255, 99, 71, 0.5);">...</h1>
<h1 style="background-color:hsla(9, 100%, 64%, 0.5);">...</h1>

RGB value format:
rgb(red, green, blue): range from 0 to 255
#rrggbb: Hexadecimal value: 00-ff(0-255)
hsl(hue, saturation, lightness):hsl(0, 100%, 50%)
rgba(red, green, blue, alpha):rgba(255, 99, 71, 0)
hsla(blue, saturation, lightness, alpha):hsla(9, 100%, 64%, 0.4)

Styling with CSS

CSS can be added in 3 ways:

inline CSS:
it is used to apply a unique style to a single HTML element.
1
<h1 style="color:blue;">This is a Blue Heading</h1>
internal CSS:
it is used to define a style for a single HTML page.
Internal CSS is defined in the section of an HTML page, within a

ML-Algo

Posted on 2018-05-09 | Edited on 2018-05-30

JAVA Design Pattern

Posted on 2018-05-08 | Edited on 2018-06-08

1 Singleton

One of the simplest design patterns in Java. This pattern involves a single class which is responsible to create an object while making sure that only single object gets created.

Step 1:
Create a Singleton Class:
SingleObject.java

public class SingleObject {

   //create an object of SingleObject
   private static SingleObject instance = new SingleObject();

   //make the constructor private so that this class cannot be
   //instantiated
   private SingleObject(){}

   //Get the only object available
   public static SingleObject getInstance(){
      return instance;
   }

   public void showMessage(){
      System.out.println("Hello World!");
   }
}

Step 2:
Get the only object from the singleton class:
SingletonPatternDemo.java

public class SingletonPatternDemo {
   public static void main(String[] args) {

      //illegal construct
      //Compile Time Error: The constructor SingleObject() is not visible
      //SingleObject object = new SingleObject();

      //Get the only object available
      SingleObject object = SingleObject.getInstance();

      //show the message
      object.showMessage();
   }
}

Step 3:
Output:

1	Hello World!

2 Factory pattern

1.
Create an interface: Shape.java

1
2
3

public interface Shape {
   void draw();
}

Create concrete classes implementing the same interface: Rectangle.java

public class Rectangle implements Shape {

   @Override
   public void draw() {
      System.out.println("Inside Rectangle::draw() method.");
   }
}

Square.java

public class Square implements Shape {

   @Override
   public void draw() {
      System.out.println("Inside Square::draw() method.");
   }
}

Circle.java

public class Circle implements Shape {

   @Override
   public void draw() {
      System.out.println("Inside Circle::draw() method.");
   }
}

Create a Factory to generate object of concrete class based on given information: ShapeFactory.java

public class ShapeFactory {

   //use getShape method to get object of type shape
   public Shape getShape(String shapeType){
      if(shapeType == null){
         return null;
      }		
      if(shapeType.equalsIgnoreCase("CIRCLE")){
         return new Circle();

      } else if(shapeType.equalsIgnoreCase("RECTANGLE")){
         return new Rectangle();

      } else if(shapeType.equalsIgnoreCase("SQUARE")){
         return new Square();
      }

      return null;
   }
}

Use the Factory to get object of concrete class by passing an information such type: FactoryPatternDemo.java

public class FactoryPatternDemo {

   public static void main(String[] args) {
      ShapeFactory shapeFactory = new ShapeFactory();

      //get an object of Circle and call its draw method.
      Shape shape1 = shapeFactory.getShape("CIRCLE");

      //call draw method of Circle
      shape1.draw();

      //get an object of Rectangle and call its draw method.
      Shape shape2 = shapeFactory.getShape("RECTANGLE");

      //call draw method of Rectangle
      shape2.draw();

      //get an object of Square and call its draw method.
      Shape shape3 = shapeFactory.getShape("SQUARE");

      //call draw method of circle
      shape3.draw();
   }
}

Output

1
2
3

Inside Circle::draw() method.
Inside Rectangle::draw() method.
Inside Square::draw() method.

3 Iterator Pattern

Create interfaces: Iterator.java

public interface Iterator {
   public boolean hasNext();
   public Object next();
}

Container.java

1
2
3

public interface Container {
   public Iterator getIterator();
}

Create concrete class implementing the Container interface. This class has inner class NameIterator implementing the Iterator interface: NameRepository.java

public class NameRepository implements Container {
   public String names[] = {"Robert" , "John" ,"Julie" , "Lora"};

   @Override
   public Iterator getIterator() {
      return new NameIterator();
   }

   private class NameIterator implements Iterator {

      int index;

      @Override
      public boolean hasNext() {

         if(index < names.length){
            return true;
         }
         return false;
      }

      @Override
      public Object next() {

         if(this.hasNext()){
            return names[index++];
         }
         return null;
      }		
   }
}

Use the NameRepository to get iterator and print names: IteratorPatternDemo.java

public class IteratorPatternDemo {

   public static void main(String[] args) {
      NameRepository namesRepository = new NameRepository();

      for(Iterator iter = namesRepository.getIterator(); iter.hasNext();){
         String name = (String)iter.next();
         System.out.println("Name : " + name);
      } 	
   }
}

Output:

Name : Robert
Name : John
Name : Julie
Name : Lora

4 Chain of Responsibilities 责任链模式（有求必应）

通过client要求，逐级传递找到责任人，最后作出决断并传回client。
client不需要知道谁处理的，只要知道结果。
击鼓传花：击鼓人：client；花：request；传花的人：request handler
购房折扣申请：
责任人：CEO-Vice President-Manager-Sales
买房者：

definition：将接受者对象连成一条链，并在链上传递request，知道有一个接受者对象处理它。此方式避免senders与receivers之间的coupling
Imgur
Example Code

缺点：耗时，内存占用大（创建的对象只用了一部分）

5 Strategy 策略模式

将可变部分抽象成算法接口，由客户选择算法。
提供多个方法作为选项。

Big Data Course Note

Posted on 2018-05-07

PPT 1

Big data: the amount of data just beyond technology’s capability to store, manage and process efficiently.

Types of Data:
structured:
usually has defined length and format;
Ex: numbers, dates, strings

unstructured:
doesn’t follow a specific format;
Ex: documents, images, video, audio, sets of data

semistructured:
falls between the other two;
may not conform to a fixed schema, but may be self-defining;
Ex: RDF, XML, Linked data

The situation where our most difficult problem is not how to store the data, but how to process it in meaningful ways.

Top 5 advantages of successfully managing Big Data:

Improving overall agency efficiency
Improving speed/accuracy of decision making
Ability ot forecast events
Ease of identifying opportunities for savings
Greater understanding of citizens needs

PPT 2

Graph theory:

Maximum flow

PPT 3

Analytics:analytics are a suite of tools for processing data to achieve actionable intelligence to support decision-making processes.

Big Data issues affecting analytics:
–Volume
–Velocity
–Variety
–Veracity
–Value

PPT 4

Statistical Machine Learning

Unsupervised:
K-means Clustering
Principal Component Analysis (PCA): Dimension reducation

Normalization:reshape value range from 0 to 1

PPT 5

Classification:
K-Nearest Neighbor
Linear Regression Model
Logistic Regression Model
Generalized Linear Model:
3 essential parts:
–Error distribution
–Link function
–Variance function

Support Vector Machines (SVM)

PPT 6

SQL:Select, Insert, Update, Delete statements
Data Definition:
–Schema defined at the start
–Triggers to respond to Insert, Update and Delete
–Alter and Drop
–Security and Access Control

Transactions: Properties
–Atomic
–Consistent
–Isolated
–Durable

NoSQL:
CAP not ACID:
–Consistency:all nodes see the same data at the same time.
–Availability:a guarantee that every request receieves a response about whether it was successful or failed. All clients always read and write data
–Partitioning: system continues to operate despite arbitrary message loss or failure of part of the system.
–Multiple entry points
Imgur
Imgur

K/V stores

Column stores:
–Data is stored across rows rather than in columns as in an RDBMS
–Read faster than write

Ex: Cassandra, HBase(Based on Google’s Big Table)

Document stores:
–Key document stores: document can be seen as value part, so we consider this model as a super set of Key-Value
–Document can be represented in many ways, but XML, JSON and BSON are typical representations.

Ex:MongoDB, CouchDB

Graph DBs
Graph databases are built with nodes, relationships between nodes and the properties of nodes.

Case: Cassandra
-Both Key-Value and Column store;
-High availability;
-Eventual consistency;
-Incremental scalability

Architecture
-Implements a peer-to-peer distribution system across nodes
-Each node is independent but interconnected with all other nodes
-Each node can accept read and write requests, regradless of where the data is located in the cluster
-When a node goes down, read/write requests can be served from other nodes

Case: MongoDB
Goals:
-Scale horizontally over commodity hardware
-Resolve problems that don’t distribute well
-Use in-memory processing wherever possible
-Auto-sharding built-in
-Dynamically add/remove capacity with no downtime
-Query contents as well as structure

JSON style documents with dynamic schemas;
No predefined schema;
Stores documents in BSON(binary format)
Imgur

Imgur

All indexes in MongoDB are B-Tree indexes.

Queries:
Imgur

CRUE Operations:
Create:

1
2
3

db.collection.insert(<document>)
db.collection.save(<document>)
db.collection.update(<document>)

Read:

1 2	db.collection.find(<query>, <projection>) db.collection.findOne(<query>, <projection>)

Update:

1	db.collection.update(<query>, <update>, <options>)

Delete:

1	db.collection.remove(<query>, <justOne>)

Imgur

Blog in Relational DB
Imgur

Blog in Document DB
Imgur

Query:
Imgur

PPT 7

Hadoop
Written in Java
3 core components: HDFS; Yarn(Scheduler); MPP(Massive Parallel Processing)/MR(MapReduce)(Processing modules)

HDFS:
Imgur

Communications:
HDFS Communication protocols are layered on the top of the TCP/IP protocol.

Pig

Hive

PPT 8

Spark
Computations in-memory

PPT 9

Graph Databases
Allow us to store entities and relationships between these entities.

It is a DBMS that supports CRUD operations:
-Normally optimized for OLTP

NoSQL-categories

Posted on 2018-05-04 | Edited on 2018-05-07

Column-oriented

Traditional relational database is row-oriented. Each row has a row id and every field store in one table.

When searching in row-oriented database, each row will be traversed, no matter a certain column data is needed or not. If you search a person whose birth month is September. Row-oriented database will search this table from left to right then from top to down.

Index the target column can promote search speed, but index each column cause extra overload and databse will traverse all columns to search data.

Column-oriented database will stores each column seperately. It’s very fast to search when the data amount is small.

This design looks like adding index for each row. It sacrifices space and writing more index to speed up searching ability. Index maps row ID to data, but column-based database maps data to rowID. Like the picture above, it is easy to search who like archery. This is an inverted index.

When to use row-oriented and when to use column-oriented? Column-based database is easy to add one new column, but is difficult to add one row. Therefore, row-oriented database is bettern than column-based on transaction. Because it achieves real-time update. It is commonly used in marketing business.

Column-oriented database has advantage of data analysis, like sum of values. Batch processing can adopt column-oriented database at night, and it supports quick search and MapReduce aggregation.

Example: HBase, Cassandra

K/V stores

This is the easiest store in NoSQL databases.

the complex k/v paire is like:

Example: Redis, Amazon Dynamo

Document-oriented stores

Store based on K/V pair, but structure more complex.

Xinxin Tang

63 posts

44 tags