Internet Intranet CIS536 Class 4 Web Server Technology

Internet / Intranet CIS-536 Class 4 Web Server Technology HTTP Protocol Log Files Brandeis University Internet/Intranet Spring 2000

Class 4 Agenda Discuss Homework Overview of Web Servers and Server Technology HTTP The Protocol For Communication Between Web Browser and Server Log Files 2

Web Servers A Basic Web Server is Just a File Server Client Requests a File via HTTP Protocol Server Delivers the File via HTTP Protocol Server Maps URL to a Subdirectory Web Server Needs Appropriate Permissions to Access Files/Directories Supports Non-HTTP Protocols FTP, Gopher, etc. A Web Server is Not HTML Specific Typically Identifies a Filetype by Extension Or Directory Where File Exists 3

Additional Common Web Server Features Additional Security Beyond That Provided by O/S Scripting Ability to Dynamically Create a Web Page Run a Program Instead of Returning a File (CGI) Return the Program Output as the Requested File Administration Log Files Performance Monitoring 4

Advanced Web Server Features Virtual Hosting Allow Multiple URL’s to Map to Same Computer Performance Optimization Caching Reliability Scalability Proxy Servers (For Security and Performance) Fetch Documents That are on Other Computers Cache Them Locally Allows for Easy Scalability Multiple Proxy Servers Can Cache Documents From One Source Computer Embedded Scripting Server Side Includes Custom Scripting Languages Server API 5

Web Servers – Added Functionality Database Connectivity SQL, My. SQL Directory Listings Icons, etc. Built-In Search Engines Built-In Image. Map Handling Multimedia Support Session Emulation Streaming Multimedia Advanced Security Encrypted HTTP S-HTTP (Secure HTTP) – Commerce. Net SSL (Secure Sockets Layer) - Netscape Web Server “Add-Ons” CGI Substitutes / CGI Optimizations Cold Fusion 6

Web Server History All Web Servers Have a Common Root httpd (NCSA) UNIX Orientation Many Features are Essentially UNIX Features Apache Website (O’Reilly) Netscape Enterprise Server Microsoft Internet Information Server A Slew of Others 7

Apache UNIX Origins – Now Ported to NT Evolved From httpd Freeware Typical UNIX Application Public Source Code Many Defaults, Conventions BUT: All is Configurable No GUI Interface Configured via Scripts, Shell Commands, Config Files Various “Flavors” Many Optional Features API Apache. SSL 8

IIS / Netscape Microsoft IIS Not Strictly Derived From httpd/Apache Windows NT However: Functionally Very Similar to Apache Emulates Many UNIX Conventions E. g. Forward Slashes Configuration via GUI Personal Web Server Peer Web Server Netscape Multi-Platform UNIX is Preferred Platform Less “Open” Than Apache More Secure? 9

UNIX File Structure Forward Slashes (/) to Separate Filenames, Directories Case Sensitive File Names Windows is Not No Limit on Filename Size / Extensions are by Convention Root is “/” User Home Directory is: “~/” Symbolic Links / Aliases Directories Can Be Spread Over Multiple Drives Can Create Non-Hierarchical Structure File Permissions Read, Write, Execute Separate Permissions for Owner, Group, All Directories are Special Cases of Files Execute Permissions = Able to Browse Directory 10

Web Server Configuration Directory Structure Virtual Document Tree Access to User Directories UNIX: ~user Symbolic Links Be Careful: May Link You Out of Directory Structure Case Sensitivity Ownership Access Server is a Process Started by a User. Has the Permissions of the User Who Started It. Default Documents Allow Directory Browsing Scripting Who is Allowed to Run Scripts? How are Scripts Identified? 11

Web Server File Access Control / Security Directory O/S Level Security IP, Domain Level Security Spoofing Directory Access. htaccess Microsoft Front-Page Extensions Encryption S-HTTP Web Protocols Only SSL TCP/IP Level V 1. 0 – V 2. X : Security Holes Found, Fixed V 3. 0 Is Current Uses Port 443 Microsoft PCT Response to Holes in SSL 2. 0 Now Use SSL 12

Server Administration Need Sysadmin and O/S Expertise Lots of “Holes” Gotchas Whenever Scripts are Allowed FTP Who is Allowed to Change Documents? Who is Allowed to Change Server Configuration? How do They Get Access? Direct Access Remote Access (e. g. FTP) Log Files Accessibility Directory Structure Management 13

HTTP The Protocol For Requesting and Delivering Web Pages Not Restricted to Returning HTML Files Client Server Model Request / Reponse TCP/IP Protocol Using Port 80 Supports Other Ports, Can Be Run Over Other Protocols “Replaced” FTP as the Primary Method For Internet File Transfer Stateless Uses MIME Format to Encapsulate Data Message Structure Similar to SMTP Mail Messages Message Header (metadata) Message Body (data) Separated From Header by a Blank Line Browser Only Displays Body, Not Header No Restrictions on Message Size / Format (as with SMTP) 14

HTTP Versions HTTP 1. 0 - Commonly Used Version HTTP 1. 1 Formalizes Many Extensions to Version 1. 0 Supports Persistent Connections Supports Compression/Decompression Supports Virtual Hosting Single Server With Multiple IP Addresses Supports Multiple Languages Supports Byte Range Transfers Useful For Re-Sending Interrupted Data Transfers Similar to Process Used By XMODEM, etc. 15

HTTP OVERVIEW HTTP Request Client (Browser) HTTP Response Web Server File System HTML CGI HTML Server Application 16

HTTP Commands Simple Structure Main Methods GET <URI> HTTP/1. 0 Request the File Specified By the URL URI is URL Without Protocol/Port HEAD Request the HTTP Header Information Only Don’t Return the File Itself POST Sends Data to The Server Typically Data From a Form Defined, But Not Widely Implemented PUT DELETE LINK UNLINK 17

Common HTTP Header Fields Additional “Parameters” to the HTTP Commands Used in HTTP Requests: Accept Lists the MIME Types That Client Can Accept E. g. Accept text/plain, text/html or Accept * Accept-Charset Lists Accepted Character Sets That Client Can Accept ASCII, ISO-8859 -1 Are Assumed Accept-Encoding Accept-Language Authorization Basic – User. Name: Password (Base 64 Encoding) Cookie From E-mail Address of Requesting User Not Typically Used For Privacy Reasons Primarily Used By Automated Clients (e. g. Bots) 18

Common HTTP Header Fields (2) Host Virtual Host – One Server Handles Multiple Sites If-Modified-Since Only Return Data if it Has Been Modified Since This Date Pragma General Purpose For “Additional” Headers Not in Standard Referrer The URL That Referred One to This URL User-Agent Name/Version of the HTTP Client Used in HTTP Responses: Allow Lists the Available Commands Supported by Server Content-Encoding Allows for Passing Data in Compressed Formats Content-Language Describes the Natural Language of the Intended Audience 19

Common HTTP Header Fields (3) Content-Length Size of the Message Body Content-Type The MIME Type For the Data Date Expires HTTP Clients Should Not Cache Data After This Date Last-Modified Location Used For Redirection MIME-Version Pragma E. g. no-cache Retry-After When Server is Unavailable. Info On When to Try Back Server Name/Version of the HTTP Server 20

Common HTTP Header Fields (4) Title Descriptive Title of the File WWW-Authenticate When Authorization Denied, Tells Client Which Methods of Authentication are Supported HTTP Status Codes Returned By the Server In First Line of Response Informational (100 -199) Successful (200 -299) Redirection (300 -399) Location in HTTP Header Specifies Redirection Client Error (400 -499) Server Error (500 -599) 21

Common Status Values 200 201 204 300 – OK – Created (Post Request Was Fulfilled) - No Content (OK. Nothing For Client to Display - Multiple Choices Requested Resource Available From Multiple Locations. List of Locations Returned in the Response. 301 - Moved Permanently 302 - Moved Temporarily 304 - Not Modified Document Hasn’t Been Modified Since If-Modified Since Date 400 - Bad Request 401 – Unauthorized 403 - Forbidden 404 – Not Found 500 – Internal Server Error 501 – Not Implemented (Server Does Not Support This. Request) 502 – Bad Gateway (Invalid Response From Server) 503 – Service Unavailable 22

Cookies Are Name Value Pairs Stored by the Client Passed in the HTTP Header Cookies Have Associated Expiration Session (Default) Date / Time Associated With a URL Path, Not a Page! Allows Passing Parameters Between Web Pages Thus Cookies are Used to Provide State Information to a Stateless Protocol 23

Web Server HTTP Functionality Content Negotiation Choose From Several Different Formats Based on Request Language Negotiation Choose From Versions of Same Document Based on Request Support for HTTP-Put, HTTP-Delete Keep-Alive As-Is Server Doesn’t Add HTTP Headers Allows You to Create Specific Behavior Redirect to Another Site Never Saved in Browser’s Cache 24

Some Definitions Hits Each HTTP Request is a Hit Accessing a Web Page May Result in Multiple Hits E. g. Each Graphic is a Hit Page Views Accessing a Single Web Page is a Page View E. g. Typing in a URL or Clicking on a Link Visits A Single Client’s Visit to Your Entire Site (Session) May Include Multiple Page Views What Constitutes a Second Visit From the Same Client? Why is This Important? Terms are Sometimes Used Interchangeably and Improperly Compare Apples to Apples Important for Commercial Web Sites Advertising is Based on Site Access Typically Sold on Page View Basis 25

Server Log Files Many Variations to Web Server Log File Formats Four Log Files Access (Transfer) Log Each Hit is Recorded User, Date/Time, HTTP Request, etc. Error Log Date/Time, Error Referrer Log Referring Page, Destination Page Agent (User) Log Client’s Browser Clearly a Need for Standardization Linking the Four Log Files Together 26

Common Log Format Host IP Address (or Hostname) of Client Some Servers Perform Lookup of IP Address RFC 931 HTTP Request: From Seldom Used. Authuser HTTP Request: Authorization User. Name if Username Authorization is Required Time Stamp HTTP Response: Date E. g. [ 10/Jun/1998: 14: 23: 34 -0700] Request The Actual HTTP Request E. g. GET /index. htm HTTP/1. 1 27

Common Log Format (2) Status The HTTP Response Status Code Transfer Volume HTTP Response: Content-Length 28

Extended Log File Format Seven Common Log Format Fields Plus Referrer HTTP Request: Referrer User Agent HTTP Request: User-Agent Identifies Browser Other Common Fields Cookies Can Help Identify Users 29

Issues Client vs. User Typically Don’t Have User Level Information Only Record IP Address of Computer Used For Access If Fixed IP Address For a Single User’s Machine This Can Identify the User Dynamically Assigned IP Addresses Identifies the Overall Domain (e. g. AOL. com) Proxy Servers All Client’s Have IP Address of Proxy Server Multiple “Sessions” at Same Time Impossible to Have Truly Accurate Information Log File Analysis Software Has Algorithms to Identify Page Views, Visits Client Level Caching Affects Logs “ISP” Level Caching Affects Logs E. g. AOL Maintains a Cache No Requirement for Clients, ISPs to Follow Expiration Info 30

Log File Maintenance on Server Log Files Grow Rapidly Log Files Compress Very Nicely Server Configurable Generate Daily/Weekly/Monthly Logs Maintenance Scripts to Cleanup Log Files Compress Archive Cycle E. g. Maintain Current Months Files 31

Log File Analysis Big Business Bread and Butter of Sites Driven By Advertising Revenue Evaluation Factors Log File Formats Supported Ability to Link Multiple Logs How Log Files are Accessed (e. g. via FTP) Display Methodology E. g. Available Via Web Pages Lookup Capabilities E. g. Map User-Agent to Browser E. g. Resolve IP Addresses to Domains, Regions Level of Analysis E. g. Calculating Visits, Return Visitors Configurability Drill-Down Capabilities Enterprise Capabilities Ability to Manage Multiple Sites 32

Log File Analysis Options Important to Understand the Core Log Files Log File Analysis Programs Make Some Assumptions Freeware Commercial Service Bureaus 33

Resources HTTP Server Comparison http: //webcompare. internet. com/chart. htm Apache Server www. apache. org Website Server http: //website. ora. com Microsoft IIS http: //www. microsoft. com/NTWorkstation/downloa ds/Recommended/Service. Packs/NT 4 Opt. Pk/Default. asp 34