We are currently building a Map manually based on the two fields that are returned by a named JPA query because JPA 2.1 only provides a getResultList() method:
#NamedQuery{name="myQuery",query="select c.name, c.number from Client c"}
HashMap<Long,String> myMap = new HashMap<Long,String>();
for(Client c: em.createNamedQuery("myQuery").getResultList() ){
myMap.put(c.getNumber, c.getName);
}
But, I feel like a custom mapper or similar would be more performant since this list could easily be 30,000+ results.
Any ideas to build a Map without iterating manually.
(I am using OpenJPA, not hibernate)
Returning a Map result using JPA Query getResultStream
Since the JPA 2.2 version, you can use the getResultStream Query method to transform the List<Tuple> result into a Map<Integer, Integer>:
Map<Integer, Integer> postCountByYearMap = entityManager.createQuery("""
select
YEAR(p.createdOn) as year,
count(p) as postCount
from
Post p
group by
YEAR(p.createdOn)
""", Tuple.class)
.getResultStream()
.collect(
Collectors.toMap(
tuple -> ((Number) tuple.get("year")).intValue(),
tuple -> ((Number) tuple.get("postCount")).intValue()
)
);
Returning a Map result using JPA Query getResultList and Java stream
If you're using JPA 2.1 or older versions but your application is running on Java 8 or a newer version, then you can use getResultList and transform the List<Tuple> to a Java 8 stream:
Map<Integer, Integer> postCountByYearMap = entityManager.createQuery("""
select
YEAR(p.createdOn) as year,
count(p) as postCount
from
Post p
group by
YEAR(p.createdOn)
""", Tuple.class)
.getResultList()
.stream()
.collect(
Collectors.toMap(
tuple -> ((Number) tuple.get("year")).intValue(),
tuple -> ((Number) tuple.get("postCount")).intValue()
)
);
Returning a Map result using a Hibernate-specific ResultTransformer
Another option is to use the MapResultTransformer class provided by the Hibernate Types open-source project:
Map<Number, Number> postCountByYearMap = (Map<Number, Number>) entityManager.createQuery("""
select
YEAR(p.createdOn) as year,
count(p) as postCount
from
Post p
group by
YEAR(p.createdOn)
""")
.unwrap(org.hibernate.query.Query.class)
.setResultTransformer(
new MapResultTransformer<Number, Number>()
)
.getSingleResult();
The MapResultTransformer is suitable for projects still running on Java 6 or using older Hibernate versions.
Avoid returning large result sets
The OP said:
But, I feel like a custom mapper or similar would be more performant
since this list could easily be 30,000+ results.
This is a terrible idea. You never need to select 30k records. How would that fit in the UI? Or, why would you operate on such a large batch of records?
You should use query pagination as this will help you reduce the transaction response time and provide better concurrency.
There is no standard way to get JPA to return a map.
see related question: JPA 2.0 native query results as map
Iterating manually should be fine. The time to iterate a list/map in memory is going to be small relative to the time to execute/return the query results. I wouldn't try to futz with the JPA internals or customization unless there was conclusive evidence that manual iteration was not workable.
Also, if you have other places where you turn query result Lists into Maps, you probably want to refactor that into a utility method with a parameter to indicate the map key property.
You can retrieve a list of java.util.Map.Entry instead.
Therefore the collection in your entity should be modeled as a Map:
#OneToMany
#MapKeyEnumerated(EnumType.STRING)
public Map<PhoneType, PhoneNumber> phones;
In the example PhoneType is a simple enum, PhoneNumber is an entity. In your query use the ENTRY keyword that was introduced in JPA 2.0 for map operations:
public List<Entry> getPersonPhones(){
return em.createQuery("SELECT ENTRY(pn) FROM Person p JOIN p.phones pn",java.util.Map.Entry.class).getResultList();
}
You are now ready to retrieve the entries and start working with it:
List<java.util.Map.Entry> phoneEntries = personDao.getPersonPhoneNumbers();
for (java.util.Map.Entry<PhoneType, PhoneNumber> entry: phoneEntries){
//entry.key(), entry.value()
}
If you still need the entries in a map but don't want to iterate through your list of entries manually, have a look on this post Convert Set<Map.Entry<K, V>> to HashMap<K, V> which works with Java 8.
This works fine.
Repository code :
#Repository
public interface BookRepository extends CrudRepository<Book,Id> {
#Query("SELECT b.name, b.author from Book b")
List<Object[]> findBooks();
}
service.java
List<Object[]> list = bookRepository.findBooks();
for (Object[] ob : list){
String key = (String)ob[0];
String value = (String)ob[1];
}
link https://codereview.stackexchange.com/questions/1409/jpa-query-to-return-a-map
Map<String,Object> map = null;
try {
EntityManager entityManager = getEntityManager();
Query query = entityManager.createNativeQuery(sql);
query.setHint(QueryHints.RESULT_TYPE, ResultType.Map);
map = (Map<String,Object>) query.getSingleResult();
}catch (Exception e){ }
List<Map<String,Object>> list = null;
try {
EntityManager entityManager = getEntityManager();
Query query = entityManager.createNativeQuery(sql);
query.setHint(QueryHints.RESULT_TYPE, ResultType.Map);
list = query.getResultList();
}catch (Exception e){ }
JPA v2.2
Though I am late here, but if someone reaches here for solution, here is my custom working solution for multiple selected columns with multiple rows:
Query query = this.entityManager.createNativeQuery("SELECT abc, xyz, pqr,...FROM...", Tuple.class);
.
.
.
List<Tuple> lst = query.getResultList();
List<Map<String, Object>> result = convertTuplesToMap(lst);
Implementation of convertTuplesToMap():
public static List<Map<String, Object>> convertTuplesToMap(List<Tuple> tuples) {
List<Map<String, Object>> result = new ArrayList<>();
for (Tuple single : tuples) {
Map<String, Object> tempMap = new HashMap<>();
for (TupleElement<?> key : single.getElements()) {
tempMap.put(key.getAlias(), single.get(key));
}
result.add(tempMap);
}
return result;
}
in case java 8
there built in entry "CustomEntryClass"
since return is stream, then caller function (repoistory layer) must have #Transactional(readonly=true|false) annotation, otherwithe exception will be thrown
make sure you will use full qualified name of class CustomEntryClass...
#Query("select new CustomEntryClass(config.propertyName, config.propertyValue) " +
"from ClientConfigBO config where config.clientCode =:clientCode ")
Stream<CustomEntryClass<String, String>> getByClientCodeMap(#Param("clientCode") String clientCode);
With custom result class and a bit of Guava, this is my approach which works quite well:
public static class SlugPair {
String canonicalSlug;
String slug;
public SlugPair(String canonicalSlug, String slug) {
super();
this.canonicalSlug = canonicalSlug;
this.slug = slug;
}
}
...
final TypedQuery<SlugPair> query = em.createQuery(
"SELECT NEW com.quikdo.core.impl.JpaPlaceRepository$SlugPair(e.canonicalSlug, e.slug) FROM "
+ entityClass.getName() + " e WHERE e.canonicalSlug IN :canonicalSlugs",
SlugPair.class);
query.setParameter("canonicalSlugs", canonicalSlugs);
final Map<String, SlugPair> existingSlugs =
FluentIterable.from(query.getResultList()).uniqueIndex(
new Function<SlugPair, String>() {
#Override #Nullable
public String apply(#Nullable SlugPair input) {
return input.canonicalSlug;
}
});
using java 8 (+) you can get results as a list of array object (each column will from select will have same index on results array) by hibernate entity manger, and then from results list into stream, map results into entry (key, value), then collect them into map of same type.
final String sql = "SELECT ID, MODE FROM MODES";
List<Object[]> result = entityManager.createNativeQuery(sql).getResultList();
return result.stream()
.map(o -> new AbstractMap.SimpleEntry<>(Long.valueOf(o[0].toString()), String.valueOf(o[1])))
.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
The easiest and simplest way worked for me is:
String[] columns = {"id","name","salary","phone","address", "dob"};
String query = "SELECT id,name,salary,phone,address,dob from users ";
List<Object[]> queryResp = em.createNativeQuery(query).getResultList();
List<Map<String,String>> dataList = new ArrayList<>();
for(Object[] obj : queryResp) {
Map<String,String> row = new HashMap<>(columns.length);
for(int i=0; i<columns.length; i++) {
if(obj[i]!=null)
row.put(columns[i], obj[i].toString());
else
row.put(columns[i], "");
}
dataList.add(row);
}
Please refer, JPA 2.0 native query results as map
In your case in Postgres, it would be something like,
List<String> list = em.createNativeQuery("select cast(json_object_agg(c.number, c.name) as text) from schema.client c")
.getResultList();
//handle exception here, this is just sample
Map map = new ObjectMapper().readValue(list.get(0), Map.class);
Kindly note, I am just sharing my workaround with Postgres.
How about this ?
#NamedNativeQueries({
#NamedNativeQuery(
name="myQuery",
query="select c.name, c.number from Client c",
resultClass=RegularClient.class
)
})
and
public static List<RegularClient> runMyQuery() {
return entityManager().createNamedQuery("myQuery").getResultList();
}
Related
I am using janus graph deployed in embedded mode. And i am using java. And firing following queries
--> g.V().has('deleted',false).valueMap() //slow
--> g.V().has('deleted',false).toList() //slow
the above query takes time after implementing composite indexes also
--> g.V().has('deleted',false) //fast
above query is fast and use implemented composite index created on deleted key.
Also tried the same using java code
List<Vertex> list = new ArrayList<>();
DateTime dt1 = new DateTime();
JanusGraphQuery<? extends JanusGraphQuery> query = GraphClient.getJGraph().query();
Iterator iterator = query.has("deleted",false).vertices().iterator();
while(iterator.hasNext()) {
Vertex next = (Vertex) iterator.next();
getPropertyMapByVertex(next); //time consuming to convert vertex into Map.
list.add(next);
}
public static Map<String, Object> getPropertyMapByVertex(Vertex vertex) {
Map<String, Object> propertyMap = new HashMap<>();
try {
if (vertex != null) {
Iterator<VertexProperty<Object>> properties = vertex.properties();
if (properties != null) {
while (properties.hasNext()) {
Property<Object> property = properties.next();
propertyMap.put(property.key(), property.value());
}
propertyMap.put(GraphConstants.VERTEX_ID, vertex.id());
}
}
} catch (Exception e) {
logger.error("Exception in getPropertyMapByVertex : {}", ExceptionUtils.getStackTrace(e));
}
return propertyMap;
}
do we have some way to fast "getPropertyMapByVertex" method or any other way to fire query and get data quickly using java.
I haven't been following the development progress in JanusGraph, but if the multiQuery API is still available, you should give it a try:
List<Vertex> vertices = g.V().has('deleted',false).toList();
graph.multiQuery().addAllVertices(vertices).properties();
That's untested code written from memory, ultimately it may look a bit different.
However, since you mention that g.V().has('deleted',false).toList() is slow, but g.V().has('deleted',false) is fast, I'm not sure if you really understand what's going on. The latter statement by itself does absolutely nothing, while the former statement actually fetches all the vertices.
Just started learning Dapper. I have an ADO.NET background. Using a demo I downloaded, I can insert/delete data from a webform into a MySql table just fine. This, however, I have searched all morning on.
In retrieving a single row from the db by ID, it doesn't return a LIST<>, it seems to be just an object (using code from the demo I downloaded). The query works, I get the object back. It has the fields: "ProductID, Description and Price".
The only way I could get the values to those three fields was like this:
System.Reflection.PropertyInfo pi = Product.GetType().GetProperty("ProductID");
System.Reflection.PropertyInfo desc = Product.GetType().GetProperty("Description");
System.Reflection.PropertyInfo price = Product.GetType().GetProperty("Price");
int _ProductID = (int)(pi.GetValue(Product, null));
string _Description = (string)(desc.GetValue(Product, null));
decimal _Price = (decimal)(price.GetValue(Product, null));
This works and gets the correct values for the three fields.
I'm used to looping through DataTables, but I just think there is probably a better way to get those values.
Is this the correct way to do this or am I missing something? I did actually read documentation and mess with this all morning before asking, too.
Some of the things I looked at seem to be very complex. I thought Dapper was supposed to simplify things.
OK, Thanks Marc. It was difficult for me to see what was supposed to be in the Dapper class files and what was supposed to be in my code behind. The original demo way of getting a product by ID had the query as .FirstOrDefault();
I changed everything to return a List<> and it all worked. I'm sure my ADO.NET is showing, but this works. In Dapper class files:
public List<Product> ProductAsList(int Id)
{
return this._db.Query<Product>("SELECT * FROM Cart_product WHERE ProductID=#Id", new { Id = Id }).**ToList()**;
}
This is just getting one row that matched the ProductID.
In page codebehind:
protected void CartItemAdd(string ProductId) // passing it the selected ProductID
{
var results = cartservice.ProductAsList(Convert.ToInt32(ProductId));
// returns that one row using Dapper ProductAsList(ProductId)
int _ProductId = 0;
string Description = string.Empty;
decimal Price = 0;
// Loop through the list and get the value of each item:
foreach (Product obj in results)
{
_ProductId = obj.ProductID;
Description = obj.Description;
Price = obj.Price;
}
// Using Dapper to insert the selected product into the shopping cart (table):
String UserName = "jbanks";
cartitem = new CartItem();
cartitem.ProductID = _ProductId;
cartitem.Quantity = 1;
cartitem.Description = Description;
cartitem.Price = Price;
cartitem.Created = DateTime.Now;
cartitem.CreatedBy = UserName;
result = cartservice.AddCartItem(cartitem);
if (result)
{
lblMessage.Text = string.Empty;
lblMessage.Text = "Successfully added a cart item";
}
}
}
It does indeed look up the product from one table and insert a selected item into another table.
Thanks again!
The main Query<T> API returns an IEnumerable<T>, which often will be a List<T>; the AsList<T>() extension method can get it back to a list without a copy, but either way: they are just T, for whatever T you asked for. If you asked for Query<Product>, then: they should be Product instances:
var results = connection.Query<Product>(someSql, someArgs); // perhaps .AsList()
foreach (Product obj in results) { // "var obj" would be fine here too
// now just use obj.ProductID, obj.Description and obj.Price
}
If that didn't work: check that you used the <T> version of Query. There is a non-generic variant too, which returns dynamic. Frankly, you should almost always use the <T> version.
Note: I'm assuming that somewhere you have something like
class Product {
public int ProductID {get;set;}
public string Description {get;set;}
public decimal Price {get;set;}
}
Currently, we are using the Document API in OrientDB version 2.2. Let us suppose we have a class Company and a class Employee. Let's suppose we are interested in all Companies with at least one employee having a name from an arbitrary list. Employees are defined as LINKEDLISTs in our Company schema.
Our query would look smth like this:
select from Company where employees in (select from Employee where name in ["John", "Paul"])
Currently we have defined the following two indexes:
Company.employees (index on the employee links (their #rid)) -> dictionary hash index and
Employee.name -> notunique index
When executing the above query with explain we see that only the second index Employee.name is used, since we did not define the above indexes as a compound index. AS far as I could understand compound indexes across different classes like in our case are not supported in Orient 2.x.
Queries like this:
select from Company let $e = select from employees where name in ["John", "Paul"] where employees in $e
do not solve our problem either.
Searching across different blogs revealed two suggestions so far:
trying to define a compound index via inheritance by introducing a parent class on employee and company and defining the above two indexes on that
https://github.com/orientechnologies/orientdb/issues/5069
bundle the two queries in a batch scrip like this:
https://github.com/orientechnologies/orientdb/issues/6684
String cmd = "begin\n";
cmd += "let a = select from Employees where name " + query + "\n";
cmd += "let b = select from Company where employees in $a\n";
cmd += "COMMIT\n";
cmd += "return $b";
Suggestion 1 did not work for us.
Suggestion 2. worked. Both indexes have been used in each separate query, but then we ran into the next limitation of Orient. Batch scripts seem to be executed only synchronously, meaning that we can only get the results as a list all at once and not one by one in a lazy fashion, which in our case is a NO GO due to the memory overhead.
One naive workaround we tried is as follows:
public class OCommandAsyncScript extends OCommandScript implements OCommandRequestAsynch{
public OCommandAsyncScript(String sql, String cmd) {
super(sql, cmd);
}
#Override
public boolean isAsynchronous() {
return true;
}
private void containsAtLeastOne(final #Nonnull ODatabaseDocumentTx documentTx,
final #Nonnull Consumer<Company> matchConsumer,
final #Nonnull String queryText
) throws TimeoutException {
String cmd = "begin\n";
cmd += "let a = select from Employee where name " + queryText + "\n";
cmd += "let b = select from Company where employees in $a\n";
cmd += "COMMIT\n";
cmd += "return $b";
final OCommandHandler resultListener = new OCommandHandler(documentTx, (document -> {
final Company companies = document2model(document);
matchConsumer.accept(company);
}));
OCommandAsyncScript request = new OCommandAsyncScript("sql", cmd);
request.setResultListener(resultListener);
documentTx.command(request).execute();
...
}
}
public class OCommandHandler implements OCommandResultListener {
private final ODatabaseDocumentTx database;
private final Consumer<ODocument> matchConsumer;
public OCommandHandler(
final #Nonnull ODatabaseDocumentTx database,
final #Nonnull Consumer<ODocument> matchConsumer
) {
this.database = database;
this.matchConsumer = matchConsumer;
}
#Override
public boolean result(Object iRecord) {
if (iRecord != null) {
final ODocument document = (ODocument) iRecord;
/*
Result handler might be asynchronous, if document is loaded in a lazy mode,
database will be queries to fetch various fields. Need to activate it on the current thread.
*/
database.activateOnCurrentThread();
matchConsumer.accept(document);
}
return true;
}
...
}
The approach of defining a custom OCommandAsyncScript did not work unfortunately. When debugging the OStorageRemote class of Orient it seems that no partial results could be read, Here the respective extract from the source code:
public Object command(final OCommandRequestText iCommand) {
....
try {
OStorageRemote.this.beginResponse(network, session);
List<ORecord> temporaryResults = new ArrayList();
boolean addNextRecord = true;
byte status;
if(asynch) {
while((status = network.readByte()) > 0) {
ORecord record = (ORecord)OChannelBinaryProtocol.readIdentifiable(network);
if(record != null) {
switch(status) {
case 1:
if(addNextRecord) {
addNextRecord = iCommand.getResultListener().result(record);
database.getLocalCache().updateRecord(record);
}
break;
case 2:
if(record.getIdentity().getClusterId() == -2) {
temporaryResults.add(record);
}
database.getLocalCache().updateRecord(record);
}
}
}
}
}
Network.readbyte() is always null, hence no records could be fetched at all.
Is there any other workaround how we could execute a sql script in asynchronus mode and retrieve results in a lazy fashion preventing the generation of large lists on our application side?
I have my dynamo db table as follows:
HashKey(xyz) ,RangeKey(timestamp)
Now for each hash key i have set of range key.
Now i want to query based on set of hashkey and i want only most recent value correspoding to that to be fetched .I dont want to do in memory sorting and then picking most recent version.
Can i do this in any way?
Use case is that i will do a bulkget and pass set of hashkey (say 100) , so i want to get one record for each hashkey
You (currently) can't set constraints on a batch get. See http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/API_BatchGetItems.html
However, for single hash keys, you can set the direction using ScanIndexForward. See http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/API_Query.html for information.
Sample java code:
new QueryRequest().withTableName("table-name")
.withScanIndexForward(false)
.withLimit(1)
.withHashKeyValue(new AttributeValue().withS("hash-key"));
It will not be very efficient though, as you will need to make this call 100 times.
Use ScanIndexForward(true for ascending and false for descending) and can also limit the result using setLimit value of Query Expression.
Please find below the code where used QueryPage for finding the single record.
public EventLogEntitySave fetchLatestEvents(String id) {
EventLogEntitySave entity = new EventLogEntitySave();
entity.setId(id);
DynamoDBQueryExpression<EventLogEntitySave> queryExpression = new DynamoDBQueryExpression<EventLogEntitySave>().withHashKeyValues(entity);
queryExpression.setScanIndexForward(false);
queryExpression.withLimit(1);
queryExpression.setLimit(1);
List<EventLogEntitySave> result = dynamoDBMapper.queryPage(EventLogEntitySave.class, queryExpression).getResults();
System.out.println("size of records = "+result.size() );
result.get(0);
}
#DynamoDBTable(tableName = "PROD_EA_Test")
public class EventLogEntitySave {
#DynamoDBHashKey
private String id;
private String reconciliationProcessId;
private String vin;
private String source;
}
public class DynamoDBConfig {
#Bean
public AmazonDynamoDB amazonDynamoDB() {
String accesskey = "";
String secretkey = "";
//
// creating dynamo client
BasicAWSCredentials credentials = new BasicAWSCredentials(accesskey, secretkey);
AmazonDynamoDB dynamo = new AmazonDynamoDBClient(credentials);
dynamo.setRegion(Region.getRegion(Regions.US_WEST_2));
return dynamo;
}
I'm trying to select all the rows out of a database using entity framework for manipulation before they're sent to the form
var ptx = [modelname].[tablename]();
ptx.[tablename].Select(????)
what goes in the ????
I used the entitydatasource and it provide everything I needed for what I wanted to do.
_repository.[tablename].ToList();
Entity Framework has one beautiful thing for it, like :
var users = context.Users;
This will select all rows in Table User, then you can use your .ToList() etc.
For newbies to Entity Framework, it is like :
PortalEntities context = new PortalEntities();
var users = context.Users;
This will select all rows in Table User
How about:
using (ModelName context = new ModelName())
{
var ptx = (from r in context.TableName select r);
}
ModelName is the class auto-generated by the designer, which inherits from ObjectContext.
You can use this code to select all rows :
C# :
var allStudents = [modelname].[tablename].Select(x => x).ToList();
You can simply iterate through the DbSet context.tablename
foreach(var row in context.tablename)
Console.WriteLn(row.field);
or to evaluate immediately into your own list
var allRows = context.tablename.ToList();
If it's under a async method then use ToListAsync()
public async Task<List<DocumentTypes>> GetAllDocumentTypes()
{
var documentTypes = await _context.DocumentTypes.ToListAsync();
return documentTypes;
}
Old post I know, but using Select(x => x) can be useful to split the EF Core (or even just Linq) expression up into a query builder.
This is handy for adding dynamic conditions.
For example:
public async Task<User> GetUser(Guid userId, string userGroup, bool noTracking = false)
{
IQueryable<User> queryable = _context.Users.Select(x => x);
if(!string.IsNullOrEmpty(userGroup))
queryable = queryable.Where(x => x.UserGroup == userGroup);
if(noTracking)
queryable = queryable.AsNoTracking();
return await queryable.FirstOrDefaultAsync(x => x.userId == userId);
}
Here is a few ways to do it (Just assume I'm using Dependency Injection for the DbConext)
public class Example
{
private readonly DbContext Context;
public Example(DbContext context)
{
Context = context;
}
public DbSetSampleOne[] DbSamples { get; set; }
public void ExampleMethod DoSomething()
{
// Example 1: This will select everything from the entity you want to select
DbSamples = Context.DbSetSampleOne.ToArray();
// Example 2: If you want to apply some filtering use the following example
DbSamples = Context.DbSetSampleOne.ToArray().Where(p => p.Field.Equals("some filter"))
}
You can use:
ptx.[tablename].Select( o => true)