While OpenFOAM 10 (release date July 2022) is located on OpenFOAM.org and GitHub,
OpenFOAM® v2212 (published December 2022) is from OpenFOAM.com and on GitLab.
Both versions sharing an unofficial Wikipedia representation on Unofficial OpenFOAM wiki.
Is there a difference on results from each versions implemented solvers running and if so, what are tasks for providing compatibility between these newest release versions (also considering source included tutorials)?
This is an age old question (openfoam.org vs openfoam.com). The direct answer can be found in the links below.
https://www.cfd-online.com/Forums/openfoam/197150-openfoam-com-versus-openfoam-org-version-use.html
https://www.reddit.com/r/OpenFOAM/comments/o6spq5/openfoamorg_versus_openfoamcom/
Basically, both are two forks of the same software, with some code added or removed. Community contribution is higher in OpenFOAM.org's version and some advanced solvers are added in OpenFOAM.com's version.
They are very similar in most aspects and differ only in very advanced solvers / simulations. By the time people start to understand those differences, they can choose the version for themselves.
Also, note that ESI-OpenFOAM can and will contain code from OpenFOAM.org, but not the other way around. The code added in ESI-OpenFOAM stays only there. It is one of the reasons it is used more in industries.
For basic usage or learning, both versions are nearly equal. I generally stick with the OpenFOAM.org's version as it is used in more academia (from personal experiences) and it is more accessible (the documentation and tutorials) (personal opinion).
To find what exact code differs for both versions, we have to see the commit history and examine solvers manually. I'll keep on adding to this answer as I find more stuff (asking friends and professors).
TL;DR i'm currently creating a cross-platform mobile news aggregator, which will identify news articles from different publishers, but about the same topic, e.g. a celebrity passes away.
I believe I found an appropriate journal that can guide me through the steps 'Document Clustering with grouping and chaining algorithms'.
(https://www.aclweb.org/anthology/I05-1025.pdf)
However many of the steps are confusing me such as:
1) Document clustering
2) Grouping and chaining algorithms
3) Understanding equations such as the one below that I'll need to compute.
Any help on the matter, or a brief description of the steps would be greatly appreciated.
Thanks for the help.
I'm also interested in any experts in this field, and would love to use your knowledge as qualitative evidence for my project. If you'd be be up for it please DM, or drop a comment. Thanks again!
I have two datasets. One with details of contracts and other with details of organizations. For eg: One dataset has details- Company name, description, company type. Other datasets has details- Contract name, Contract description, CPV code.
I want an algorithm that can 1) given a company can we find the top 10 contracts that are most closely related or potentially interesting to this company.
2. Or given a contract can we find the companies most likely to bid or win the contract.
This might be a one off, real time algorithm to match one row of the first dataset to a best match cluster in the second dataset.
Is it possible to do this type of row by row cross matching in two different datasets? Is it possible to use text descriptions for this kind of matching?
It would be of great help if someone has code examples. Thank you.
I am also attaching example datasets here.
Company data
Contract data
Your question is effectively "Will someone do ~10K worth of data science for me for free?" What you are looking for is a recommender system and what seems more specifically to be a content based filtering system. In order for these to work, you are going to have to look at your two datasets and develop features that can be used to quantitatively describe the contracts and the clients. If you have information about previous contracts the organizations were interested in you can use a hybrid algorithm that incorporates aspects of collaborative filtering.
R has a package recommenderlab that can help you to work on these types of problems. I haven't used it, but skimming over it, it seems to be solid. If you are wanting something a little more plug and play though with fewer options, I would recommend checking out AzureML. It uses GUI interfaces to help guide users through the data science process including a recommender tutorial. You may also be able to use some of their text classifier tutorial to help engineer features from your fields containing free form text.
Best of luck.
I am trying to implement the RaptorQ Forward Error Correction Scheme in java as specified here:
https://datatracker.ietf.org/doc/html/draft-ietf-rmt-bb-fec-raptorq-04#section-5.3.3
The core of the problem is actually to execute gaussian elimination on a matrix A in a smart way to be fast.
The matrix A is composed of submatrices, among others these are G_LDPC,1 and G_LDPC,2.
(Generator matrices for Low Density Parity Checks)
On page 22 in section "5.3.3.3. Pre-coding relationships" it is stated that this matrices can be decuced from the code snippet on the same page.
My Problem: I am not able to derive the structure of these two submatrices from the code snipped.
Does someone see how to do that, or how the structure looks like?
Thanks for any kind of help!
Max
I'm also trying to implement RaptorQ, and ran into this exactly same problem. My suggestion is this book:
Raptor Codes (Foundations and Trends(R) in Communications and Information Theory) [Paperback]
Amin Shokrollahi (Author), Michael Luby (Author)
It has a better explanation on constructing the constraint matrix in section 3.3.3 (I'd quote it, but I don't have it digital).
#Max anyway we can chat or you can share your RFC5053 implementation? I really could use someone familiar with these difficulties to talk to and share some doubts/ideas.
After being stuck with the problem, I decided to implement the Raptor codec according to RFC 5053 as described here:
https://www.rfc-editor.org/rfc/rfc5053
This is actually the predecessor version of RaptorQ.
The general working principle seems to be the same, but it is less optimized and therefore has worse properties, especially in sense of reception efficiency.
But on the other hand it was less complex and more intuitive to me, and therefore I was able to code a working implementation in Java.
And after all, I have to admit that I'm very astonished by the capabilities of the created codec!
With the deeper understanding gained during coding the RFC 5053 implementation I was probably also able to realize the RaptorQ codec now.
Could you recommend a training path to start and become very good in Information Extraction. I started reading about it to do one of my hobby project and soon realized that I would have to be good at math (Algebra, Stats, Prob). I have read some of the introductory books on different math topics (and its so much fun). Looking for some guidance. Please help.
Update: Just to answer one of the comment. I am more interested in Text Information Extraction.
Just to answer one of the comment. I am more interested in Text Information Extraction.
Depending on the nature of your project, Natural language processing, and Computational linguistics can both come in handy -they provide tools to measure, and extract features from the textual information, and apply training, scoring, or classification.
Good introductory books include OReilly's Programming Collective Intelligence (chapters on "searching, and ranking", Document filtering, and maybe decision trees).
Suggested projects utilizing this knowledge: POS (part-of-speech) tagging, and named entity recognition (ability to recognize names, places, and dates from the plain text). You can use Wikipedia as a training corpus since most of the target information is already extracted in infoboxes -this might provide you with some limited amount of measurement feedback.
The other big hammer in IE is search, a field not to be underestimated. Again, OReilly's book provides some introduction in basic ranking; once you have a large corpus of indexed text, you can do some really IE tasks with it. Check out Peter Norvig: Theorizing from data as a starting point, and a very good motivator -maybe you could reimplement some of their results as a learning exercise.
As a fore-warning, I think I'm obligated to tell you, that information extraction is hard. The first 80% of any given task is usually trivial; however, the difficulty of each additional percentage for IE tasks are usually growing exponentially -in development, and research time. It's also quite underdocumented -most of the high-quality info is currently in obscure white papers (Google Scholar is your friend) -do check them out once you've got your hand burned a couple of times. But most importantly, do not let these obstacles throw you off -there are certainly big opportunities to make progress in this area.
I would recommend the excellent book Introduction to Information Retrieval by Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze. It covers a broad area of issues which form a great and up-to-date (2008) basis for Information Extraction and is available online in full text (under the given link).
I would suggest you take a look at the Natural Language Toolkit (nltk) and the NLTK Book. Both are available for free and are great learning tools.
You don't need to be good at math to do IE just understand how the algorithm works, experiment on the cases for which you need an optimal result performance, and the scale with which you need to achieve target accuracy level and work with that. You are basically working with algorithms and programming and aspects of CS/AI/Machine learning theory not writing a PhD paper on building a new machine-learning algorithm where you have to convince someone by way of mathematical principles why the algorithm works so I totally disagree with that notion. There is a difference between practical and theory - as we all know mathematicians are stuck more on theory then the practicability of algorithms to produce workable business solutions. You would, however, need to do some background reading both books in NLP as well as journal papers to find out what people found from their results. IE is a very context-specific domain so you would need to define first in what context you are trying to extract information - How would you define this information? What is your structured model? Supposing you are extracting from semi and unstructured data sets. You would then also want to weigh out whether you want to approach your IE from a standard human approach which involves things like regular expressions and pattern matching or would you want to do it using statistical machine learning approaches like Markov Chains. You can even look at hybrid approaches.
A standard process model you can follow to do your extraction is to adapt a data/text mining approach:
pre-processing - define and standardize your data to extraction from various or specific sources cleansing your data
segmentation/classification/clustering/association - your black box where most of your extraction work will be done
post-processing - cleansing your data back to where you want to store it or represent it as information
Also, you need to understand the difference between what is data and what is information. As you can reuse your discovered information as sources of data to build more information maps/trees/graphs. It is all very contextualized.
standard steps for: input->process->output
If you are using Java/C++ there are loads of frameworks and libraries available you can work with.
Perl would be an excellent language to do your NLP extraction work with if you want to do a lot of standard text extraction.
You may want to represent your data as XML or even as RDF graphs (Semantic Web) and for your defined contextual model you can build up relationship and association graphs that most likely will change as you make more and more extractions requests. Deploy it as a restful service as you want to treat it as a resource for documents. You can even link it to taxonomized data sets and faceted searching say using Solr.
Good sources to read are:
Handbook of Computational Linguistics and Natural Language Processing
Foundations of Statistical Natural Language Processing
Information Extraction Applications in Prospect
An Introduction to Language Processing with Perl and Prolog
Speech and Language Processing (Jurafsky)
Text Mining Application Programming
The Text Mining Handbook
Taming Text
Algorithms of Intelligent Web
Building Search Applications
IEEE Journal
Make sure you do a thorough evaluation before deploying such applications/algorithms into production as they can recursively increase your data storage requirements. You could use AWS/Hadoop for clustering, Mahout for large scale classification amongst others. Store your datasets in MongoDB or unstructured dumps into jackrabbit, etc. Try experimenting with prototypes first. There are various archives you can use to base your training on say Reuters corpus, tipster, TREC, etc. You can even check out alchemy API, GATE, UIMA, OpenNLP, etc.
Building extractions from standard text is easier than say a web document so representation at pre-processing step becomes even more crucial to define what exactly it is you are trying to extract from a standardized document representation.
Standard measures include precision, recall, f1 measure amongst others.
I disagree with the people who recommend reading Programming Collective Intelligence. If you want to do anything of even moderate complexity, you need to be good at applied math and PCI gives you a false sense of confidence. For example, when it talks of SVM, it just says that libSVM is a good way of implementing them.
Now, libSVM is definitely a good package but who cares about packages. What you need to know is why SVM gives the terrific results that it gives and how it is fundamentally different from Bayesian way of thinking ( and how Vapnik is a legend).
IMHO, there is no one solution to it. You should have a good grip on Linear Algebra and probability and Bayesian theory. Bayes, I should add, is as important for this as oxygen for human beings ( its a little exaggerated but you get what I mean, right ?). Also, get a good grip on Machine Learning. Just using other people's work is perfectly fine but the moment you want to know why something was done the way it was, you will have to know something about ML.
Check these two for that :
http://pindancing.blogspot.com/2010/01/learning-about-machine-learniing.html
http://measuringmeasures.com/blog/2010/1/15/learning-about-statistical-learning.html
http://measuringmeasures.com/blog/2010/3/12/learning-about-machine-learning-2nd-ed.html
Okay, now that's three of them :) / Cool
The Wikipedia Information Extraction article is a quick introduction.
At a more academic level, you might want to skim a paper like Integrating Probabilistic Extraction Models and Data Mining to Discover Relations and Patterns in Text.
Take a look here if you need enterprise grade NER service. Developing a NER system (and training sets) is a very time consuming and high skilled task.
This is a little off topic, but you might want to read Programming Collective Intelligence from O'Reilly. It deals indirectly with text information extraction, and it doesn't assume much of a math background.