Misplaced Pages

#542457

60-405: With a narrower scope, reproducibility has been defined in computational sciences as having the following quality: the results should be documented by making all data and code available in such a way that the computations can be executed again with identical results. In recent decades, there has been a rising concern that many published scientific results fail the test of reproducibility, evoking

120-408: A random effects model , also called a variance components model , is a statistical model where the model parameters are random variables . It is a kind of hierarchical linear model , which assumes that the data being analysed are drawn from a hierarchy of different populations whose differences relate to that hierarchy. A random effects model is a special case of a mixed model . Contrast this to

180-714: A system as a potential source of data, an experiment as a process of extracting data from a system by exerting it through its inputs and a model ( M ) for a system ( S ) and an experiment ( E ) as anything to which E can be applied in order to answer questions about S . A computational scientist should be capable of: Substantial effort in computational sciences has been devoted to developing algorithms, efficient implementation in programming languages, and validating computational results. A collection of problems and solutions in computational science can be found in Steeb, Hardy, Hardy, and Stoop (2004). Philosophers of science addressed

240-407: A first difference will remove any time invariant components of the model. Two common assumptions can be made about the individual specific effect: the random effects assumption and the fixed effects assumption. The random effects assumption is that the individual unobserved heterogeneity is uncorrelated with the independent variables. The fixed effect assumption is that the individual specific effect

300-406: A greater understanding of city dynamics and help prepare for the coming urbanization . In financial markets , huge volumes of interdependent assets are traded by a large number of interacting market participants in different locations and time zones. Their behavior is of unprecedented complexity and the characterization and measurement of the risk inherent to this highly diverse set of instruments

360-446: A high frequency current to light gas-filled lamps from over 25 miles (40 km) away without using wires . In 1904 he built Wardenclyffe Tower on Long Island to demonstrate means to send and receive power without connecting wires. The facility was never fully operational and was not completed due to economic problems, so no attempt to reproduce his first result was ever carried out. Other examples which contrary evidence has refuted

420-495: A multidisciplinary doctorate Ph.D. program in Computational Sciences and Informatics starting from 1992. The School of Computational and Integrative Sciences, Jawaharlal Nehru University (erstwhile School of Information Technology ) also offers a vibrant master's science program for computational science with two specialties: Computational Biology and Complex Systems . Variance component In statistics ,

480-544: A periodic loading condition, "the probability is (say) 90% that the number of cycles at failure (Nf) will be in the interval N1<Nf<N2". Cities are massively complex systems created by humans, made up of humans, and governed by humans. Trying to predict, understand and somehow shape the development of cities in the future requires complex thinking and computational models and simulations to help mitigate challenges and possible disasters. The focus of research in urban complex systems is, through modeling and simulation, to build

540-467: A plane, automobile body distortions in a crash, the motion of stars in a galaxy, an explosive device, etc. Such programs might create a 'logical mesh' in computer memory where each item corresponds to an area in space and contains information about that space relevant to the model. For example, in weather models , each item might be a square kilometer; with land elevation, current wind direction, humidity, temperature, pressure, etc. The program would calculate

600-468: A primary source such as surveys, field observations, experimental research, or obtaining data from an existing source. Data processing involves the processing and review of the raw data collected in the first stage, and includes data entry, data manipulation and filtering and may be done using software. The data should be digitized and prepared for data analysis. Data may be analysed with the use of software to interpret or visualise statistics or data to produce

660-487: A renewal of internal concerns about irreproducible results (see the entry on replicability crisis for empirical results on success rates of replications). Researchers showed in a 2006 study that, of 141 authors of a publication from the American Psychological Association (APA) empirical articles, 103 (73%) did not respond with their data over a six-month period. In a follow-up study published in 2015, it

SECTION 10

#1732884980543

720-632: A reproducibility or replication crisis . The first to stress the importance of reproducibility in science was the Anglo-Irish chemist Robert Boyle , in England in the 17th century. Boyle's air pump was designed to generate and study vacuum , which at the time was a very controversial concept. Indeed, distinguished philosophers such as René Descartes and Thomas Hobbes denied the very possibility of vacuum existence. Historians of science Steven Shapin and Simon Schaffer , in their 1985 book Leviathan and

780-490: A scientific fact, and in practice for establishing scientific authority in any field of knowledge. However, as noted above by Shapin and Schaffer, this dogma is not well-formulated quantitatively, such as statistical significance for instance, and therefore it is not explicitly established how many times must a fact be replicated to be considered reproducible. Replicability and repeatability are related terms broadly or loosely synonymous with reproducibility (for example, among

840-478: A sequence of smaller steps that are combined so that the intermediate outputs from one step directly feed as inputs into the next step. Version control should be used as it lets the history of the project be easily reviewed and allows for the documenting and tracking of changes in a transparent manner. A basic workflow for reproducible research involves data acquisition, data processing and data analysis. Data acquisition primarily consists of obtaining primary data from

900-483: A standard aptitude test are ascertained. Let Y i j {\displaystyle Y_{ij}} be the score of the j {\displaystyle j} -th pupil at the i {\displaystyle i} -th school. A simple way to model this variable is where μ {\displaystyle \mu } is the average test score for the entire population. In this model U i {\displaystyle U_{i}}

960-577: A wide domain in the former is used in CSE (e.g., certain algorithms, data structures, parallel programming, high-performance computing), and some problems in the latter can be modeled and solved with CSE methods (as an application area). Algorithms and mathematical methods used in computational science are varied. Commonly applied methods include: Historically and today, Fortran remains popular for most applications of scientific computing. Other programming languages and computer algebra systems commonly used for

1020-425: Is a scientific discipline concerned with the formulation, calibration, numerical solution, and validation of mathematical models designed to predict specific aspects of physical events, given initial and boundary conditions, and a set of characterizing parameters and associated uncertainties. In typical cases, the predictive statement is formulated in terms of probabilities. For example, given a mechanical component and

1080-457: Is correlated with the independent variables. If the random effects assumption holds, the random effects estimator is more efficient than the fixed effects model. Suppose m {\displaystyle m} large elementary schools are chosen randomly from among thousands in a large country. Suppose also that n {\displaystyle n} pupils of the same age are chosen randomly at each selected school. Their scores on

1140-485: Is heavily used in scientific computing to find solutions of large problems in a reasonable amount of time. In this framework, the problem is either divided over many cores on a single CPU node (such as with OpenMP ), divided over many CPU nodes networked together (such as with MPI ), or is run on one or more GPUs (typically using either CUDA or OpenCL ). Computational science application programs often model real-world changing conditions, such as weather, airflow around

1200-622: Is low or no incentives for researchers to share their data, and authors would have to bear the costs of compiling data into reusable forms. Economic research is often not reproducible as only a portion of journals have adequate disclosure policies for datasets and program code, and even if they do, authors frequently do not comply with them or they are not enforced by the publisher. A Study of 599 articles published in 37 peer-reviewed journals revealed that while some journals have achieved significant compliance rates, significant portion have only partially complied, or not complied at all. On an article level,

1260-486: Is measured repeatedly in different laboratories to assess the variability of the measurements. Then, the standard deviation of the difference between two values obtained within the same laboratory is called repeatability. The standard deviation for the difference between two measurement from different laboratories is called reproducibility . These measures are related to the more general concept of variance components in metrology . The term reproducible research refers to

SECTION 20

#1732884980543

1320-414: Is successful. The terms reproducibility and replicability sometimes appear even in the scientific literature with reversed meaning, as different research fields settled on their own definitions for the same terms. In chemistry, the terms reproducibility and repeatability are used with a specific quantitative meaning. In inter-laboratory experiments, a concentration or other quantity of a chemical substance

1380-413: Is the school-specific random effect : it measures the difference between the average score at school i {\displaystyle i} and the average score in the entire country. The term W i j {\displaystyle W_{ij}} is the individual-specific random effect, i.e., it's the deviation of the j {\displaystyle j} -th pupil's score from

1440-598: Is to gain understanding through the analysis of mathematical models implemented on computers . Scientists and engineers develop computer programs and application software that model systems being studied and run these programs with various sets of input parameters. The essence of computational science is the application of numerical algorithms and computational mathematics . In some cases, these models require massive amounts of calculations (usually floating-point ) and are often executed on supercomputers or distributed computing platforms. The term computational scientist

1500-465: Is typically based on complicated mathematical and computational models . Solving these models exactly in closed form, even at a single instrument level, is typically not possible, and therefore we have to look for efficient numerical algorithms . This has become even more urgent and complex recently, as the credit crisis has clearly demonstrated the role of cascading effects going from single instruments through portfolios of single institutions to even

1560-494: Is used to describe someone skilled in scientific computing. Such a person is usually a scientist, an engineer, or an applied mathematician who applies high-performance computing in different ways to advance the state-of-the-art in their respective applied disciplines in physics, chemistry, or engineering. Computational science is now commonly considered a third mode of science , complementing and adding to experimentation / observation and theory (see image). Here, one defines

1620-755: The CONSORT initiative, which is now part of a wider initiative, the EQUATOR Network . This group has recently turned its attention to how better reporting might reduce waste in research, especially biomedical research. Reproducible research is key to new discoveries in pharmacology . A Phase I discovery will be followed by Phase II reproductions as a drug develops towards commercial production. In recent decades Phase II success has fallen from 28% to 18%. A 2011 study found that 65% of medical studies were inconsistent when re-tested, and only 6% were completely reproducible. Hideyo Noguchi became famous for correctly identifying

1680-701: The University of Amsterdam and the Vrije Universiteit in computational science was first offered in 2004. In this program, students: ETH Zurich offers a bachelor's and master's degree in Computational Science and Engineering. The degree equips students with the ability to understand scientific problem and apply numerical methods to solve such problems. The directions of specializations include Physics, Chemistry, Biology and other Scientific and Engineering disciplines. George Mason University has offered

1740-502: The biostatistics definitions, as biostatisticians use "fixed" and "random" effects to respectively refer to the population-average and subject-specific effects (and where the latter are generally assumed to be unknown, latent variables ). Random effect models assist in controlling for unobserved heterogeneity when the heterogeneity is constant over time and not correlated with independent variables. This constant can be removed from longitudinal data through differencing, since taking

1800-471: The "third mode of discovery" (next to theory and experimentation). In many fields, computer simulation is integral and therefore essential to business and research. Computer simulation provides the capability to enter fields that are either inaccessible to traditional experimentation or where carrying out traditional empirical inquiries is prohibitively expensive. CSE should neither be confused with pure computer science , nor with computer engineering , although

1860-466: The Air-Pump , describe the debate between Boyle and Hobbes, ostensibly over the nature of vacuum, as fundamentally an argument about how useful knowledge should be gained. Boyle, a pioneer of the experimental method , maintained that the foundations of knowledge should be constituted by experimentally produced facts, which can be made believable to a scientific community by their reproducibility. By repeating

Reproducibility - Misplaced Pages Continue

1920-419: The average compliance rate was 47.5%; and on a journal level, the average compliance rate was 38%, ranging from 13% to 99%. A 2018 study published in the journal PLOS ONE found that 14.4% of a sample of public health statistics researchers had shared their data or code or both. There have been initiatives to improve reporting and hence reproducibility in the medical literature for many years, beginning with

1980-615: The average education level of a child's parents. This is a mixed model , not a purely random effects model, as it introduces fixed-effects terms for Sex and Parents' Education. The variance of Y i j {\displaystyle Y_{ij}} is the sum of the variances τ 2 {\displaystyle \tau ^{2}} and σ 2 {\displaystyle \sigma ^{2}} of U i {\displaystyle U_{i}} and W i j {\displaystyle W_{ij}} respectively. Let be

2040-525: The average for the i {\displaystyle i} -th school. The model can be augmented by including additional explanatory variables, which would capture differences in scores among different groups. For example: where S e x i j {\displaystyle \mathrm {Sex} _{ij}} is a binary dummy variable and P a r e n t s E d u c i j {\displaystyle \mathrm {ParentsEduc} _{ij}} records, say,

2100-464: The average, not of all scores at the i {\displaystyle i} -th school, but of those at the i {\displaystyle i} -th school that are included in the random sample . Let be the grand average . Let be respectively the sum of squares due to differences within groups and the sum of squares due to difference between groups. Then it can be shown that and These " expected mean squares " can be used as

2160-455: The bacterial agent of syphilis , but also claimed that he could culture this agent in his laboratory. Nobody else has been able to produce this latter result. In March 1989, University of Utah chemists Stanley Pons and Martin Fleischmann reported the production of excess heat that could only be explained by a nuclear process (" cold fusion "). The report was astounding given the simplicity of

2220-484: The basis for estimation of the "variance components" σ 2 {\displaystyle \sigma ^{2}} and τ 2 {\displaystyle \tau ^{2}} . The σ 2 {\displaystyle \sigma ^{2}} parameter is also called the intraclass correlation coefficient . For random effects models the marginal likelihoods are important. Random effects models used in practice include

2280-486: The credibility and reliability of published research. In other sciences, reproducibility is regarded as fundamental and is often a prerequisite to research being published, however in economic sciences it is not seen as a priority of the greatest importance. Most peer-reviewed economic journals do not take any substantive measures to ensure that published results are reproducible, however, the top economics journals have been moving to adopt mandatory data and code archives. There

2340-466: The desired results of the research such as quantitative results including figures and tables. The use of software and automation enhances the reproducibility of research methods. There are systems that facilitate such documentation, like the R Markdown language or the Jupyter notebook. The Open Science Framework provides a platform and useful tools to support reproducible research. Psychology has seen

2400-492: The direct management of Boyle and his assistant at the time Robert Hooke . Huygens reported an effect he termed "anomalous suspension", in which water appeared to levitate in a glass jar inside his air pump (in fact suspended over an air bubble), but Boyle and Hooke could not replicate this phenomenon in their own pumps. As Shapin and Schaffer describe, "it became clear that unless the phenomenon could be produced in England with one of

2460-480: The equipment: it was essentially an electrolysis cell containing heavy water and a palladium cathode which rapidly absorbed the deuterium produced during electrolysis. The news media reported on the experiments widely, and it was a front-page item on many newspapers around the world (see science by press conference ). Over the next several months others tried to replicate the experiment, but were unsuccessful. Nikola Tesla claimed as early as 1899 to have used

Reproducibility - Misplaced Pages Continue

2520-733: The few available options to understand such systems is by developing a multi-scale model of the system. Using information theory , non-equilibrium dynamics , and explicit simulations, computational systems theory tries to uncover the true nature of complex adaptive systems . Computational science and engineering (CSE) is a relatively new discipline that deals with the development and application of computational models and simulations, often coupled with high-performance computing , to solve complex physical problems arising in engineering analysis and design (computational engineering) as well as natural phenomena (computational science). CSE has become accepted amongst scientists, engineers and academics as

2580-423: The following research categorizations. In practical use, it is typically the application of computer simulation and other forms of computation from numerical analysis and theoretical computer science to solve problems in various scientific disciplines. The field is different from theory and laboratory experiments, which are the traditional forms of science and engineering . The scientific computing approach

2640-446: The foundations for the modern scientific practice of hypothesis testing and statistical significance , that "we may say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us statistically significant results". Such assertions express a common dogma in modern science that reproducibility is a necessary condition (although not necessarily sufficient ) for establishing

2700-405: The gene expression data in a systematic way and to guide future data collection. A major challenge here is to understand how gene regulation is controlling fundamental biological processes like biomineralization and embryogenesis . The sub-processes like gene regulation , organic molecules interacting with the mineral deposition process, cellular processes , physiology , and other processes at

2760-401: The general public), but they are often usefully differentiated in more precise senses, as follows. Two major steps are naturally distinguished in connection with reproducibility of experimental or observational studies: When new data is obtained in the attempt to achieve it, the term replicability is often used, and the new study is a replication or replicate of the original one. Obtaining

2820-634: The idea that scientific results should be documented in such a way that their deduction is fully transparent. This requires a detailed description of the methods used to obtain the data and making the full dataset and the code to calculate the results easily accessible. This is the essential part of open science . To make any research project computationally reproducible, general practice involves all data and files being clearly separated, labelled, and documented. All operations should be fully documented and automated as much as practicable, avoiding manual intervention where feasible. The workflow should be designed as

2880-658: The interconnected trading network. Understanding this requires a multi-scale and holistic approach where interdependent risk factors such as market, credit, and liquidity risk are modeled simultaneously and at different interconnected scales. Exciting new developments in biotechnology are now revolutionizing biology and biomedical research . Examples of these techniques are high-throughput sequencing , high-throughput quantitative PCR , intra-cellular imaging, in-situ hybridization of gene expression, three-dimensional imaging techniques like Light Sheet Fluorescence Microscopy , and Optical Projection (micro)-Computer Tomography . Given

2940-672: The likely next state based on the current state, in simulated time steps, solving differential equations that describe how the system operates, and then repeat the process to calculate the next state. In 2001, the International Conference on Computational Science (ICCS) was first organized. Since then, it has been organized yearly. ICCS is an A-rank conference in the CORE ranking . The Journal of Computational Science published its first issue in May 2010. The Journal of Open Research Software

3000-414: The massive amounts of complicated data that is generated by these techniques, their meaningful interpretation, and even their storage, form major challenges calling for new approaches. Going beyond current bioinformatics approaches, computational biology needs to develop new methods to discover meaningful patterns in these large data sets. Model-based reconstruction of gene networks can be used to organize

3060-488: The more mathematical aspects of scientific computing applications include GNU Octave , Haskell , Julia , Maple , Mathematica , MATLAB , Python (with third-party SciPy library ), Perl (with third-party PDL library), R , Scilab , and TK Solver . The more computationally intensive aspects of scientific computing will often use some variation of C or Fortran and optimized algebra libraries such as BLAS or LAPACK . In addition, parallel computing

SECTION 50

#1732884980543

3120-525: The original claim: Computational science Computational science , also known as scientific computing , technical computing or scientific computation ( SC ), is a division of science, and more specifically the Computer Sciences , which uses advanced computing capabilities to understand and solve complex physical problems. While this discussion typically extenuates into Visual Computation , this research field of study will typically include

3180-853: The question to what degree computational science qualifies as science, among them Humphreys and Gelfert. They address the general question of epistemology: how does one gain insight from such computational science approaches? Tolk uses these insights to show the epistemological constraints of computer-based simulation research. As computational science uses mathematical models representing the underlying theory in executable form, in essence, they apply modeling (theory building) and simulation (implementation and execution). While simulation and computational science are our most sophisticated way to express our knowledge and understanding, they also come with all constraints and limits already known for computational solutions. Problem domains for computational science/scientific computing include: Predictive computational science

3240-513: The same experiment over and over again, Boyle argued, the certainty of fact will emerge. The air pump, which in the 17th century was a complicated and expensive apparatus to build, also led to one of the first documented disputes over the reproducibility of a particular scientific phenomenon . In the 1660s, the Dutch scientist Christiaan Huygens built his own air pump in Amsterdam , the first one outside

3300-457: The same results when analyzing the data set of the original study again with the same procedures, many authors use the term reproducibility in a narrow, technical sense coming from its use in computational research. Repeatability is related to the repetition of the experiment within the same study by the same researchers. Reproducibility in the original, wide sense is only acknowledged if a replication performed by an independent researcher team

3360-402: The tissue and environmental levels are linked. Rather than being directed by a central control mechanism, biomineralization and embryogenesis can be viewed as an emergent behavior resulting from a complex system in which several sub-processes on very different temporal and spatial scales (ranging from nanometer and nanoseconds to meters and years) are connected into a multi-scale system. One of

3420-478: The two pumps available, then no one in England would accept the claims Huygens had made, or his competence in working the pump". Huygens was finally invited to England in 1663, and under his personal guidance Hooke was able to replicate anomalous suspension of water. Following this Huygens was elected a Foreign Member of the Royal Society . However, Shapin and Schaffer also note that "the accomplishment of replication

3480-425: Was dependent on contingent acts of judgment. One cannot write down a formula saying when replication was or was not achieved". The philosopher of science Karl Popper noted briefly in his famous 1934 book The Logic of Scientific Discovery that "non-reproducible single occurrences are of no significance to science". The statistician Ronald Fisher wrote in his 1935 book The Design of Experiments , which set

3540-581: Was found that 246 out of 394 contacted authors of papers in APA journals did not share their data upon request (62%). In a 2012 paper, it was suggested that researchers should publish data along with their works, and a dataset was released alongside as a demonstration. In 2017, an article published in Scientific Data suggested that this may not be sufficient and that the whole analysis context should be disclosed. In economics, concerns have been raised in relation to

3600-474: Was launched in 2012 . The ReScience C initiative, which is dedicated to replicating computational results, was started on GitHub in 2015. At some institutions, a specialization in scientific computation can be earned as a "minor" within another program (which may be at varying levels). However, there are increasingly many bachelor's , master's , and doctoral programs in computational science. The joint degree program master program computational science at

#542457