DataMelt: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
|||
Line 9: | Line 9: | ||
|field_license_type=Free&Open source | |field_license_type=Free&Open source | ||
|field_free_software_licence=GPL / GNU for non commercial use. Commercial-friendly license is available | |field_free_software_licence=GPL / GNU for non commercial use. Commercial-friendly license is available | ||
|field_last_release= | |field_last_release=2018/05/10 | ||
|field_last_version= | |field_last_version=2.2 | ||
|field_description='''DataMelt is a software environment for numeric calculations, statistics and data analysis ''' | |field_description='''DataMelt is a software environment for numeric calculations, statistics and data analysis ''' | ||
Line 20: | Line 20: | ||
DMelt creates high-quality vector-graphics images (SVG, EPS, PDF etc.) that can be included in LaTeX and other text-processing systems. | DMelt creates high-quality vector-graphics images (SVG, EPS, PDF etc.) that can be included in LaTeX and other text-processing systems. | ||
==History== | |||
DataMelt has its roots in [[particle physics]] where data mining is a primary task. It was created as [[jHepWork]] project in 2005 and it was initially written for data analysis for [[particle physics]]<ref> | |||
HEP data analysis using jHepWork and Java, arXiv:0809.0840v2, ANL-HEP-CP-08-53 preprint. CERN preprint, [https://arxiv.org/abs/0809.0840/ arXiv:0809.0840v2] | |||
</ref> using the [[Java (programming language)|Java]] software concept for [[International linear collider|International Linear Collider]] project developed at [[SLAC]]. Later versions of jHepWork were modified for general public use (for scientists, engineers, students for educational purpose) since the International Linear Collider project has stalled. In 2013, jHepWork was renamed to DataMelt and become a general-purpose community-supported project. | |||
The main source of the reference is the book "Scientific Data analysis using Jython Scripting and Java" | |||
<ref> | |||
Scientific Data analysis using Jython Scripting and Java. Book. By S.V.Chekanov, Springer-Verlag, ISBN 978-1-84996-286-5, [https://www.springer.com/us/book/9781849962865] | |||
</ref> | |||
which discusses data-analysis methods using [[Java (programming language)|Java]] and [[Jython]] scripting. | |||
Later it was also discussed in the German Java SPEKTRUM journal | |||
<ref> | |||
DataMelt – Werkbank für technisch-wissenschaftliche Berechnungen und Visualisierungen mit Java und Jython. by Rohe Klaus. Java SPEKTRUM. (in German) volume 5 (2013) 26-28 [https://www.sigs-datacom.de/fachzeitschriften/javaspektrum/archiv/artikelansicht/artikel-titel/integrationsspektrum-scavis-werkbank-fuer-technisch-wissenschaftliche-berechnungen-und-visualis.html] | |||
</ref>. | |||
The string "HEP" in the project name "jHepWork" abbreviates "High-Energy Physics". But due to a wide popularity outside this area of physics, it was renamed to [[SCaViS]] ('''S'''cientific '''C'''omputation and '''Vis'''ualization Environment). This project existed for 3 years before it was renamed to DataMelt (or, in short, DMelt). | |||
DataMelt is hosted by the jWork.ORG portal<ref> jWork.ORG Community Portal focused on Java scientific software. [http://jwork.org/main/]</ref> | |||
==Supported platforms== | |||
DataMelt runs on Windows, Linux, Mac and the [[Android (operating system)|Android]] platforms. The package for the Android is called AWork. | |||
==Documentation== | |||
DataMelt is extensively documented. In 2018, the web page of this project contained about 600 examples written in Jython, Java, Groovy, JRuby, covering a number of fields, from general mathematics to data mining and data visualization. The Java API documentation includes the description of more than 40,000 Java classes. In addition, | |||
there is a wiki documentation. The documentation includes certain restrictions for general public due to the proprietorial nature of the documentation project. | |||
==License terms== | |||
DataMelt is licensed by [[Freemium]] license. | |||
The core source code of the numerical and graphical libraries is licensed by the [[GNU General Public License]]. The interactive development environment (IDE) used by DataMelt has some restrictions for commercial usage since language files, documentation files, examples, installer, code-assist databases, interactive help are licensed by the creative-common license. Full members of the DataMelt project have several benefits, such as: the license for a commercial usage, access to the source repository, an extended help system, a user script repository and an access to the complete documentation. | |||
The commercial licenses cannot apply to source code that was imported or contributed<ref>{{cite web|url=http://jwork.org/wiki/DMelt:Dev/Contributions|title=Contributed Packages (DataMelt Manual)}}</ref> to DataMelt from other authors. | |||
==Examples== | |||
===Jython scripts=== | |||
Here is an example of how to show 2D bar graphs by reading a CVS file downloaded from the [[World Bank]] web site. | |||
<syntaxhighlight lang="python"> | |||
from jhplot.io.csv import * | |||
from java.io import * | |||
from jhplot import * | |||
d = {} | |||
reader = CSVReader(FileReader("ny.gdp.pcap.cd_Indicator_en_csv_v2.csv")); | |||
while True: | |||
nextLine = reader.readNext() | |||
if nextLine is None: | |||
break | |||
xlen = len(nextLine) | |||
if xlen < 50: | |||
continue | |||
d[nextLine[0]] = float(nextLine[xlen-2]) # key=country, value=DGP | |||
c1 = HChart("2013",800,400) | |||
#c1.setGTitle("2013 Gross domestic product per capita") | |||
c1.visible() | |||
c1.setChartBar() | |||
c1.setNameY("current US $") | |||
c1.setNameX("") | |||
c1.setName("2013 Gross domestic product per capita") | |||
name1 = "Data Source: World Development Indicators" | |||
set_value = lambda name: c1.valueBar(d[name], name, name1) | |||
set_value(name="Russia") | |||
set_value(name="Poland") | |||
set_value(name="Romania") | |||
set_value(name="Bulgaria") | |||
set_value(name="Belarus") | |||
set_value(name="Ukraine") | |||
c1.update() | |||
</syntaxhighlight> | |||
The execution of this script plots a bar chart in a separate window. The image can be saved in a number of formats. | |||
Here is another simple example which illustrates how to fill a 2D histogram and display it on a canvas. The script also creates a figure in the [[Portable Document Format|PDF]] format. | |||
This script illustrates how to glue and mix the native JAVA classes (from the package java.util) and DataMelt classes (the package jhplot) inside a script written using the Python syntax. | |||
<syntaxhighlight lang="python"> | |||
from java.util import Random | |||
from jhplot import * | |||
c1 = HPlot3D("Canvas") # create an interactive canvas | |||
c1.setGTitle("Global title") | |||
c1.setNameX("X") | |||
c1.setNameY("Y") | |||
c1.visible() | |||
c1.setAutoRange() | |||
h1 = H2D("2D histogram", 25, -3.0, 3.0, 25, -3.0, 3.0) | |||
rand = Random() | |||
for i in range(200): | |||
h1.fill(rand.nextGaussian(), rand.nextGaussian()) | |||
c1.draw(h1) | |||
c1.export("jhplot3d.eps") # export to EPS Vector Graphics | |||
</syntaxhighlight> | |||
This script can be run either using DataMelt IDE or using a stand-alone Jython after specifying classpath to DataMelt libraries. The output is shown below: | |||
[[File:Histograms_in_3D_showing_random_Gaussian_data.png|thumb|center|222px|3D histogram]] | |||
===Groovy scripts=== | |||
The same example can also be coded using the [[Groovy (programming language)|Groovy]] programming language which is supported by DataMelt. | |||
<syntaxhighlight lang="groovy"> | |||
import java.util.Random | |||
import jhplot.* | |||
c1 = new HPlot3D("Canvas") // create an interactive canvas | |||
c1.setGTitle("Global title") | |||
c1.setNameX("X") | |||
c1.setNameY("Y") | |||
c1.visible() | |||
c1.setAutoRange() | |||
h1 = new H2D("2D histogram",25,-3.0, 3.0,25,-3.0, 3.0) | |||
rand = Random() | |||
(1..200).each{ // or (0..<200).each{ | |||
// or Java: for (i=0; i<200; i++){ | |||
// if argument is required, you cann access it through "it" inside the loop: | |||
// (0..<200).each{ println "step: ${it+1}" } | |||
h1.fill(rand.nextGaussian(),rand.nextGaussian()) | |||
} | |||
c1.draw(h1); | |||
c1.export("jhplot3d.eps") // export to EPS Vector Graphics | |||
</syntaxhighlight> | |||
[[Groovy (programming language)|Groovy]] is better integrated with Java and can be a factor three | |||
faster for long loops over primitives compared to Jython. | |||
==Reviews== | |||
DataMelt and its earlier versions, SCaVis (2013-2015) and JHepWork (2005-2013), which are still available from [http://jwork.org/dmelt/index.php?id=previous DataMelt archive repository], are described in these articles: | |||
<ref>Data Analysis and Data Mining Using Java, Jython and jHepWork Blog. 2010. Oracle.com. [https://community.oracle.com/docs/DOC-982931] | |||
</ref> | |||
<ref> | |||
DataMelt – Werkbank für technisch-wissenschaftliche Berechnungen und Visualisierungen mit Java und Jython. by Rohe Klaus. Java SPEKTRUM. (in German) volume 5 (2013) 26-28 [https://www.sigs-datacom.de/fachzeitschriften/javaspektrum/archiv/artikelansicht/artikel-titel/integrationsspektrum-scavis-werkbank-fuer-technisch-wissenschaftliche-berechnungen-und-visualis.html] | |||
</ref> | |||
<ref>HEP data analysis using jHepWork and Java. Proceedings of the HERA-LHC workshops (2007-2008), DESY-CERN [https://arxiv.org/abs/0809.0840] | |||
</ref> | |||
<ref>Suitability analysis of data mining tools and methods. [https://is.muni.cz/th/255695/fi_b/]. S.Kovac, Bachelor's thesis (in English), jHepWork is reviewed on page 39-42, Masaryk University. | |||
</ref> | |||
The program was compared with other similar frameworks in these resources | |||
<ref> | |||
A Review: Comparative Study of Diverse Collection of Data Mining Tools. By S. Sarumathi, N. Shanthi, S. Vidhya, M. Sharmila. International Journal of Computer, Control, Quantum and Information Engineering. 2014; 8(6). 7. | |||
</ref> | |||
<ref>jHepWork – full-featured multiplatform data-analysis framework. [http://blog.dreamcss.com/frameworks/jhepwork-multiplatform-data-analysis-framework/] | |||
</ref> | |||
<ref>Java Applications: Weka, Svnkit, Jhepwork, Fiji, Memoranda, Livegraph, Dirsync Pro, Moneydance, Rachota Timetracker, Data Mapping Engine. de LLC Books (2010) (Redactor) [https://www.amazon.es/Java-Applications-Memoranda-Moneydance-Timetracker/dp/1156999391] | |||
</ref> | |||
<ref>A Study of Tools, Techniques, and Trends for Big Data Analytics. By R.Shireesha et al. (2016) International Journal of Advance Computing Technique and Applications (IJACTA), ISSN : 2321-4546, Vol 4, Issue 1 [http://www.ijacta.com/index.php/ojs/article/download/41/33] | |||
</ref> | |||
<ref>Comparison of Various Tools for Data Mining. By P.Kaur etc. IJERT ISSN: 2278-0181 Vol. 3 Issue 10 (2010) [[https://www.ijert.org/download/11405/comparison-of-various-tools-for-data-mining]] | |||
</ref> | |||
<ref>Advanced Web and Network Technologies, and Applications. By Heng Tao Shen et al. Springer Science & Business Media - 2006-01-09 | |||
</ref> | |||
<ref>SCaVis 2.1. Review by Pete Daniel (2014) [https://www.download3k.com/Home-Education/Science/Download-SCaVis.html] | |||
</ref> | |||
. | |||
The DataMelt (2015-), a new development of the JHepWork and SCaVis programs. | |||
Comparisons of DataMelt with other popular packages for statistical and numeric analysis are given in these resources | |||
<ref>Comparative Analysis of Information Extraction Techniques for Data Mining, by Amit Verma et al. Indian Journal of Science and Technology, Vol 9, March 2016 [http://www.indjst.org/index.php/indjst/article/download/80464/67992] | |||
</ref> | |||
<ref>Evaluation and comparison of open source software suites for data mining and knowledge discovery. A.H. Altalhi et al. Wiley Online Library (2017) [https://onlinelibrary.wiley.com/doi/full/10.1002/widm.1204] | |||
</ref> | |||
<ref>Brief Review of Educational Applications Using Data Mining and Machine Learning, [https://redie.uabc.mx/redie/article/viewFile/1305/1586], by A. Berenice Urbina Nájera, Jorgede la Calleja Mora, Redie ISSN 1607-4041. Revista Electrónica de Investigación Educativa, 19(4), 84-96 | |||
</ref> | |||
<ref>Analysis of Data Using Data Mining tool Orange. Maqsud S.Kukasvadiya et. al. [https://www.ijedr.org/papers/IJEDR1702288.pdf] (2017) IJEDR, Volume 5, Issue 2, ISSN: 2321-9939 | |||
</ref> | |||
<ref>Big Data - A Survey of Big Data Technologies. By P.Dhavalchandra, M.Jignasu, R.Amit. International Journal of Science and Technology. Volume 2, p45-50 (2016) [http://www.ijsrt.us/vol2issue1/pdf%209.pdf] | |||
</ref> | |||
. | |||
==Popularity== | |||
jHepWork, SCaVis/DatMelt are part of the software library of National Institutes of Health Library | |||
<ref>Data Sciences Workstation: SCaVis. By Lisa Federer. National Institutes of Health Library [http://nihlibrary.campusguides.com/c.php?g=102650&p=672758]</ref>, | |||
Mathematical support of Institute for Nuclear Research of Russian academy of Sciences<ref>The DataForge, Sector for Mathematical Support of Institute for Nuclear Research of Russian academy of Sciences [http://www.inr.ru/~nozik/dataforge/misc.html]</ref> and others. | |||
On a commercial site, DataMelt is provided as a service on Amazon EC2 clouds by the Miri Infotech IT Solution Provider company | |||
<ref>Miri Infotech. A Complete IT Solution Provider. [http://www.miritech.com/products/aws/datamelt.aspx DataMelt deployment]</ref>. | |||
It is difficult to judge how many users use DataMelt since download information from the main resource [http://jwork.org/dmelt] is not available. | |||
[[Sourceforge|Sourceforge]], which provides an alternative download option, quotes 300 monthly downloads [https://sourceforge.net/projects/dmelt/] (May 2018). | |||
One estimate can be done by looking at the popularity of the book | |||
<ref>Numeric Computation and Statistical Data Analysis on the Java Platform (Book). S.V.Chekanov, Springer, (2016) | |||
ISBN 978-3-319-28531-3, 700 pages, [https://www.springer.com/gp/book/9783319285290]</ref> | |||
which is an introduction to the DataMelt program. According to the | |||
Springer International, this book is top 25% most downloadable books in 2016 and 2017 in the category "Advanced Information and Knowledge Processing" | |||
<ref>Springer Book Performance Report [http://jwork.org/dmelt/data_dm/reports/BookPerformanceReport2017.pdf]</ref>. | |||
Since the publication of the book, Springer detects 26k chapter downloads until May 2018<ref>Springer download Statistics of the book "Numeric Computation and Statistical Data Analysis on the Java Platform" 2016 [http://www.bookmetrix.com/detail/book/1b56c407-cc52-49ee-9547-5ec300e18498#downloads]</ref>, about 1500 per chapter. | |||
The previous book describing jHepWork had a similar popularity <ref>Springer download Statistics of the book "Scientific Data Analysis using Jython Scripting and Java" [http://www.bookmetrix.com/detail/book/0d0c46b5-edee-4145-b112-4167b6739c11#downloads]</ref>. | |||
Bookmetrix estimates 140 readers of the DataMelt book. | |||
== References == | |||