Input output Prediction Estimation Machine Learning Linear Regression

기계학습이란? • 지도학습 세부 분류 - 훈령용 데이터에 알고리즘을 적용하여 함수를 추론, 추론된 함수를 통해 컴퓨터가 알아서 답을 찾도록 만드는 것, 지도학습은 명확한 Input과 output이 존재 예측, 추정 기계학습 (Prediction, Estimation) (Machine Learning) . Linear Regression (선형회귀). Regression Tree, Model Tree (회귀 나무). SVN (Support Vector Machine) (서포트 벡터 머신) 지도학습 (Supervised learning) . Neural Network, Deep Learning (신경망). ARIMA, Exponential Smoothing 분류 (Classification) Decision Tree (의사 결정 나무) Logistic Regression, Discriminant Analysis k-NN (k-Nearest Neighbor) Naive Bayes Classification SVM

기계학습이란? • 비지도학습 세부 분류 - 특성이 비슷한 데이터를 합쳐서 군(Group)으로 분류하는 학습 방법, 결과 정보가 없는 데이터들에 대해 특정 패턴을 찾는 것 레이블이 없는 데이터를 사용하는데, 목적은 데이터 안에서 어떤관계(Relationships)를 찾는데 있음, 즉 사람이 개입하지 않고 컴퓨 터 스스로 데이터를 훈련하는 것 패턴/구조 발견 기계학습 (Pattern/Rule) (Machine Learning) => Association Rule Analysis, Sequence Analysis => Network Analysis, Link Analysis, Graph theory => Structural Equation Modeling, Path Analysis 비지도학습 (Unsupervised learning) 그룹화 (Grouping) => k-Means Clustering, Hierarchical Clustering, Density-based Clustering, Fuzzy Clustering => SOM(Self-Organizing Map) (Dimesion Reduction) => PCA(Principal Component Analysis), Factor Analysis, SVD(Singular Value Decomposition) 영상, 이미지, 문자 => Wavelet/Fast Fourier Transformation 자원축소 (Dimesion Reduction) => Text mining

Tensorflow (텐서플로) • 텐서플로우의 자료형 Ø 상수형(Constant) tensorflow. constant(value, dtype=None, shape=None, name="Const", verify_shape=False) 1 2 3 1 value 상수 값 2 dtype 상수의 데이타 형, ex: 실수, 정수 3 shape 행렬의 차원을 정의 4 name은 이 상수의 이름을 정의 4 5

Tensorflow (텐서플로) • 텐서플로우의 자료형 e. g import tensorflow as tf def main(): s = tf. constant(10, dtype=tf. int 16, shape=(1, ), name="Constant_number", verify_shape=False) print (s) if __name__ == "__main__": main() Tensor("Constant_number: 0", shape=(1, ), dtype=int 16)

Tensorflow (텐서플로) • 그래프와 세션의 개념 s 1 = tf. constant(10, dtype=tf. int 16, shape=(1, ), name="Constant_number", verify_shape=False) s 2 = tf. constant(11, dtype=tf. int 16, shape=(1, ), name="Constant_number", verify_shape=False) s 3 = tf. constant(12, dtype=tf. int 16, shape=(1, ), name="Constant_number", verify_shape=False) d = (s 1+s 2)*s 3 s 1 s 2 s 3 + *

Tensorflow (텐서플로) • 그래프와 세션의 개념 Ø 세션(Session)을 생성하여, 그래프를 실행 s 1 = tf. constant(10, dtype=tf. int 16, shape=(1, ), name="Constant_number", verify_shape=False) s 2 = tf. constant(11, dtype=tf. int 16, shape=(1, ), name="Constant_number", verify_shape=False) s 3 = tf. constant(12, dtype=tf. int 16, shape=(1, ), name="Constant_number", verify_shape=False) d = (s 1+s 2)*s 3 sess = tf. Session() result = sess. run(d) print (result)

Tensorflow (텐서플로) • Tensor. Flow Mechanics

Tensorflow (텐서플로) 동작을 정의한 것 • Tensor. Flow Mechanics operation 그래프 operation edge : Node와 Node를 연결 operation 정의를 포함한 것 Node

Tensorflow (텐서플로) • Tensor. Flow Mechanics import tensorflow as tf Ø 위처럼 Tensor. Flow 일단 import 하면 내부적으로 default_graph_stack에 Default Graph가 생긴 다. tf. get_default_graph() 명령어로 접근 가능 graph = tf. get_default_graph() input = tf. constant(value=4. 9, dtype=tf. float 32, shape=(1, ), name="My_number 1") input 2 = tf. constant(value=4, dtype=tf. int 64, shape=(1, ), name="My_number 2") operations = graph. get_operations() for i in range(2): print (operations[i]. node_def)

name: "My_number 1" op: "Const" attr { key: "dtype" value { type: DT_FLOAT } } attr { key: "value" value { tensor { dtype: DT_FLOAT tensor_shape { dim { size: 1 } } float_val: 4. 900000095367432 } } } name: "My_number 2" op: "Const" attr { key: "dtype" value { type: DT_INT 64 } } attr { key: "value" value { tensor { dtype: DT_INT 64 tensor_shape { dim { size: 1 } } int 64_val: 4 } } }

Tensorflow (텐서플로) • #tf. constant : 말 그대로 상수입니다. import tensorflow as tf def main(): hello = tf. constant('Hello. Tensor. Flow') # Tensor("Const: 0", shape=(), dtype=string) print (hello) if __name__ == "__main__": main()

Tensorflow (텐서플로) • tf. add import tensorflow as tf def main(): v 1 = tf. constant(10) v 2 = tf. constant(32) v 3 = tf. add(v 1, v 2) print ("{} + {} = {}". format(v 1, v 2, v 3)) sess = tf. Session() print (sess. run([v 1, v 2, v 3])) # [10, 32, 42] sess. close() if __name__ == "__main__": main()

Tensorflow (텐서플로) • tf. reduce_mean(d) Ø 데이터 값의 평균 import tensorflow as tf import numpy as np def main(): d = [float(m) for m in range(10, 20)] print ("d => {}". format(d)) ave = tf. reduce_mean(d) sess = tf. Session() data = sess. run(ave) print ("ave => {}". format(data)) sess. close()

Tensorflow (텐서플로) • Placeholders import tensorflow as tf def main(): a = tf. placeholder(tf. float 32) b = tf. placeholder(tf. float 32) adder_node = a+b sess = tf. Session() print (sess. run(adder_node, feed_dict={a: 3, b: 4})) print (sess. run(adder_node, feed_dict={a: [1, 2, 3], b: [4, 5, 6]})) sess. close() if __name__ == "__main__": main()

Tensorflow (텐서플로) • tf. abs ( [+] absolute : 절댓값 ) abs( ) x, name=None • Args Ø x : type => float 32, float 64, int 32, int 64, complex 128 Ø name => A name for the operation (optional) • Returns Ø complex 64, complex 128

Tensorflow (텐서플로) • tf. abs ( [+] absolute : 절댓값 ) v = tf. constant(value=-10. 0, dtype=np. float 32) pprint(v) # <tf. Tensor 'Const: 0' shape=() dtype=float 32> sess = tf. Session() r = sess. run(v) pprint("r => {}". format(r)) positive_r = tf. abs(r) positive_r = sess. run(positive_r) pprint("positive_r => {}". format(positive_r)) sess. close()

Tensorflow (텐서플로) • tf. complex( real, imag, name=None ) Ø real : A Tensor. Must be one of the following types: float 32, float 64 Ø imag : A Tensor. Must have the same type as real Ø name: A name for the operation (optional) Ø Returns: A Tensor of type complex 64 or complex 128.

Tensorflow (텐서플로) • tf. complex import tensorflow as tf import pprint as ppr def main(): v = tf. complex(real=12. 0, imag=24. 0, name=None) print (v) # Tensor("Complex: 0", shape=(), dtype=complex 64) sess = tf. Session() complex 01 = sess. run(v) (12+24 j) pprint(complex 01) sess. close() if __name__ == "__main__": main()

Tensorflow (텐서플로) • tf. complex

Tensorflow (텐서플로) • tf. complex import tensorflow as tf import pprint as ppr def main(): t_complex 01 = tf. complex(real=23. 4, imag=45. 2, name=None) t_complex 02 = tf. complex(real=12. 23, imag=-34. 0, name=None) print ("complex 01 => {}ncomplex 02 => {}". format(t_complex 01, t_complex 02)) sess = tf. Session() Complex. V = sess. run([t_complex 01, t_complex 02]) # 반드시 리스트로 묶어라 Total. Sum = tf. add(t_complex 01, t_complex 02) s = sess. run([Total. Sum]) pprint("{} + {} = {}". format( Complex. V[0], Complex. V[1], s )) sess. close() if __name__ == "__main__": main()

Tensorflow (텐서플로) • tf. complex(real = , imag = ) # study_1 import tensorflow as tf import pprint as ppr def main(): real = tf. constant([2. 25, 3. 25]) # 실수 imag = tf. constant([1. 234, 2. 456]) # 허수 s = tf. complex(real=real, imag=imag) print ("s => {}". format(s)) sess = tf. Session() t = sess. run(s) pprint(t) sess. close() if __name__ == "__main__": main()

Tensorflow (텐서플로) • tf. accumulate_n() accumulate_n( inputs, shape=None, tensor_dtype=None, name=None ) • Args: Ø input : A list of Tensor object, each with same shape and type Ø shape : Shape of elements of inputs. Ø tensor_dtype : The type of inputs Ø name : A name for the operation (optional) • Returns Ø A Tensor of same shape and type as the elements of inputs.

Tensorflow (텐서플로) • tf. accumulate_n() Ø tf. accumulate_n([v 1, v 2], shape=[2, 2], tensor_dtype=np. int 32) import tensorflow as tf import numpy as np def main(): # 2 by 2 mat v 1 = np. array([[1, 2], [3, 4]]) # 2 by 2 mat v 2 = np. array([[10, 11], [12, 13]]) v 3 = tf. accumulate_n([v 1, v 2], shape=[2, 2], tensor_dtype=np. int 32)

Tensorflow (텐서플로) • tf. accumulate_n() def main(): vec_1 = np. array([[1, 2], [3, 4]]) vec_2 = np. array([[11, 12], [13, 14]]) T_v 1 = tf. constant(vec_1) T_v 2 = tf. constant(vec_2) T_sum = tf. accumulate_n([T_v 1, T_v 2]) print ("T_v 1 => {} , T_v 2 => {}". format(T_v 1, T_v 2)) print ("T_sum => {}". format(T_sum)) sess = tf. Session() # <- OPEN d = sess. run([T_v 1, T_v 2, T_sum]) pprint("{} + {} = {}". format(d[0], d[1], d[2])) sess. close() if __name__ == "__main__": main()

Tensorflow (텐서플로) • tf. log() log( ) x, name=None • Args: Ø x : A Tensor. Must be one of the following types: half, float 32, float 64, complex 128 Ø name : A name for the operation (optional) • Returns: Ø A Tensor. Has the same type as x.

Tensorflow (텐서플로) • tf. log() import tensorflow as tf import math import pprint as ppr def main(): s = tf. log(x=math. e, name=None) pprint(s) sess = tf. Session() re = sess. run(s) pprint ("Re : {}". format(re)) sess. close()

크롤링과 스크레이핑 • urllib. request를 이용한 다운로드 import urllib. request url = "http: //blog. koreadaily. com/_data/user/d/a/o/daotim/image/2016/9/20205533_7. jpg" save. Name = "test. png" urllib. request. urlretrieve(url, save. Name) print ("저장되었습니다. ") • import urllib. request # 해석: "urllib 패키지 내부에 있는 request 모듈" • urlretrieve(arg 1, arg 2) # arg 1 : url , arg 2 : 저장할 파일의 경로

크롤링과 스크레이핑 • urlopen() Ø "메모리 위에 데이터를 올리고, 이후에 저장" import urllib. request def main(): url = "http: //blog. koreadaily. com/_data/user/d/a/o/daotim/image/2016/9/20205533_7. jpg" savename = "test 1. png" # 다운로드 -- (*1) mem = urllib. request. urlopen(url). read() # 파일로 저장하기 -- (*2) with open(savename, mode='wb') as f: f. write(mem) print ("저장 되었습니다. ") if __name__ == "__main__": main()

크롤링과 스크레이핑 • 웹에서 데이터 추출하기 Ø 웹에서 XML 또는 HTML 등의 텍스트 기반 데이터를 다운로드 • 클라이언트 접속 정보 출력해보기 import urllib. request def main(): url = "http: //api. aoikujira. com/ip/ini" res = urllib. request. urlopen(url) data = res. read() text = data. decode("utf-8") print (text) if __name__ == "__main__": main()

Beautiful. Soup로 스크레이핑하기 • Beautiful. Soup 기본 사용법 html from bs 4 import Beautiful. Soup # 분석하고 싶은 HTML = """ <html> <head>"안녕"</head> <body> <p 1>p 1 테스트 입니다. </p 1> <p>"hello"</p> <p>"world"</p> </body> </html> """ soup = Beautiful. Soup(HTML, "html. parser") # 원하는 부분 h 1 = soup. html. body. p 1 print (h 1. string) p 1 = soup. html. body. p print (p 1. string) p 2 = p 1. next_sibling print (p 2. string) head body p 1 p "html"을 분석할때

Beautiful. Soup로 스크레이핑하기 • 실습 (1) from bs 4 import Beautiful. Soup import urllib. request url = "http: //news. naver. com/main/hotissue/read. nhn? mid=hot&sid 1=100&cid=1049580&iid=24956150&oid=056&aid=0010536366&ptype=052" HTML = urllib. request. urlopen(url) soup = Beautiful. Soup(HTML, "html. parser") print (soup. html. head. title. string)

Beautiful. Soup로 스크레이핑하기 • id로 요소를 찾는 방법 Ø Beautiful. Soup 는 루트부터 하나하나 요소를 찾는 방법 말고도 id 속성을 지정해서 요소를 찾 는 find() 메서드를 제공 from bs 4 import Beautiful. Soup import urllib. request url = "http: //news. naver. com/main/hotissue/read. nhn? mid=hot&sid 1=100&cid=1049580" "&iid=24956150&oid=056&aid=0010536366&ptype=052" html = urllib. request. urlopen(url) soup = Beautiful. Soup(html, "html. parser") title = soup. find(id="article. Title") print ("title => {}". format(title. string)) • 여러 개의 요소 추출하기 - find_all() 메서드 Ø 여러 개의 태그를 한 번에 추출하고 싶을 때는 find_all 메서드 사용

Beautiful. Soup로 스크레이핑하기 • 실습(2) from bs 4 import Beautiful. Soup import urllib. request import pprint url = "http: //news. naver. com/main/hotissue/read. nhn? mid=hot&sid 1=100&cid=1049580" "&iid=24956150&oid=056&aid=0010536366&ptype=052" html = urllib. request. urlopen(url) soup = Beautiful. Soup(html, "html. parser") # title = soup. find(id="article. Title") # print ("title => {}". format(title. string)) content = soup. find_all("link") pprint(content) for a in content: href = a. attrs['href'] text = a. string # 현재 안쪽에 내용이 없다. 출력을 해도 "None"이다 print (href)

Beautiful. Soup로 스크레이핑하기 • DOM 요소의 속성에 대해 Ø DOM (Document Object Model) 이란 XML 또는 HTML 의 요소에 접근하는 구조 from bs 4 import Beautiful. Soup soup = Beautiful. Soup( "<html>" "<head>'hello world'</head>" "<body>" "<p>'test'</p>" "</body>" "</html>", "html. parser" ) a = soup. html. body. p print (type(a. attrs)) # <class 'dict'> print (soup. prettify()) print (a. string) # 'test' <class 'dict'> <html> <head> 'hello world' </head> <body> <p> 'test' </p> </body> </html> 'test' Process finished with exit code 0

pip install bs 4 pip install requests

from bs 4 import Beautiful. Soup import requests from pprint import pprint def main(): URL = "https: //www. google. co. kr/search? " params = { "q": "뉴스공장" } response = requests. get(URL, params=params) html = response. text soup = Beautiful. Soup(html, "html. parser") h 3_data = soup. find_all(class_='r') for title in h 3_data: print (title. text) if __name__ == "__main__": main()

URL = "https: //www. google. co. kr/search? " for i in range(1, 10): params = { "q": "뉴스공장", "start": (i-1)*10 } response = requests. get(URL, params=params) html = response. text soup = Beautiful. Soup(html, "html. parser") h 3_data = soup. find_all(class_='r') for title in h 3_data: print (title. text)