Scrapy
Scrapy 0.22.2 (2014/02/14)
The last edition of this page was on: 2014/02/26
The Completion level of this page is : Low
The last edition of this page was on: 2014/02/26 The Completion level of this page is : Low
SHORT DESCRIPTION
Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing. Features
- Scrapy was designed with simplicity in mind, by providing the features you need without getting in your way
- Just write the rules to extract the data from web pages and let Scrapy crawl the entire web site for you
- Scrapy is used in production crawlers to completely scrape more than 500 retailer sites daily, all in one server
- Scrapy was designed with extensibility in mind and so it provides several mechanisms to plug new code without having to touch the framework core
- Scrapy is completely written in Python and runs on Linux, Windows, Mac and BSD
- Scrapy comes with lots of functionality built in. Check this section of the documentation for a list of them.
- Scrapy is extensively documented and has an comprehensive test suite with very good code coverage
TOOL CHARACTERISTICS
Usability
Tool orientation
Data mining type
Manipulation type
IMPORT FORMAT :
EXPORT FORMAT :
| Tool objective(s) in the field of Learning Sciences | |
|
☑ Analysis & Visualisation of data |
☑ Providing feedback for supporting instructors: |
Tool can perform:
- Data extraction of type:
- Transformation of type:
- Data analysis of type:
- Data visualisation of type: (These visualisations can be interactive and updated in "real time")
ABOUT USERS
Tool is suitable for:
Required skills:
STATISTICS:
PROGRAMMING:
SYSTEM ADMINISTRATION:
DATA MINING MODELS:
FREE TEXT
| Tool version : Scrapy 0.22.2 2014/02/14 (blank line) Developed by : Scrapy community | 
|
SHORT DESCRIPTION
Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing. Features
- Scrapy was designed with simplicity in mind, by providing the features you need without getting in your way
- Just write the rules to extract the data from web pages and let Scrapy crawl the entire web site for you
- Scrapy is used in production crawlers to completely scrape more than 500 retailer sites daily, all in one server
- Scrapy was designed with extensibility in mind and so it provides several mechanisms to plug new code without having to touch the framework core
- Scrapy is completely written in Python and runs on Linux, Windows, Mac and BSD
- Scrapy comes with lots of functionality built in. Check this section of the documentation for a list of them.
- Scrapy is extensively documented and has an comprehensive test suite with very good code coverage
TOOL CHARACTERISTICS
| Tool orientation | Data mining type | Usability |
|---|---|---|
| This tool is designed for general purpose analysis. | This tool is designed for . | Authors of this page consider that this tool is . |
| Data import format | Data export format |
|---|---|
| . | . |
| Tool objective(s) in the field of Learning Sciences | |
|
☑ Analysis & Visualisation of data |
☑ Providing feedback for supporting instructors: |
Can perform data extraction of type:
Can perform data transformation of type:
Can perform data analysis of type:
Can perform data visualisation of type:
(These visualisations can be interactive and updated in "real time")
ABOUT USER
| Tool is suitable for: | ||||
| Students/Learners/Consumers:☑ | Teachers/Tutors/Managers:☑ | Researchers:☑ | Organisations/Institutions/Firms:☑ | Others:☑ |
| Required skills: | |||
| Statistics: | Programming: | System administration: | Data mining models: |
OTHER TOOL INFORMATION
|
| Scrapy |
| Public Domain |
| Free&Open source |
| Scrapy community |
| 2014/02/14 |
| 0.22.2 |
| http://scrapy.org/ |
| Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
Features
|
| General analysis |
| Low |