play 2.3.8 sbt excluding logback - sbt

I'm having a really hard time excluding logback from my play 2.3.8 test run. I've tried many exclude rules, but nothing seems to work. I also can't find it in my dependency tree. Snippet from my sbt file:
[...]
resolvers ++= Seq(
"Typesafe repository snapshots" at "http://repo.typesafe.com/typesafe/snapshots/",
"Typesafe repository releases" at "http://repo.typesafe.com/typesafe/releases/",
"Sonatype repo" at "https://oss.sonatype.org/content/groups/scala-tools/",
"Sonatype releases" at "https://oss.sonatype.org/content/repositories/releases",
"Sonatype snapshots" at "https://oss.sonatype.org/content/repositories/snapshots",
"Sonatype staging" at "http://oss.sonatype.org/content/repositories/staging",
"Java.net Maven2 Repository" at "http://download.java.net/maven/2/",
"Twitter Repository" at "http://maven.twttr.com",
"Websudos releases" at "http://maven.websudos.co.uk/ext-release-local"
)
libraryDependencies ++= {
val phantomVersion = "1.5.0"
Seq(
"net.jpountz.lz4" % "lz4" % "1.3.0",
"org.xerial.snappy" % "snappy-java" % "1.1.1.6",
"com.websudos" %% "phantom-dsl" % phantomVersion,
"com.websudos" %% "phantom-testing" % phantomVersion % Test,
"org.scalatestplus" %% "play" % "1.2.0" % Test,
"org.cassandraunit" % "cassandra-unit" % "2.1.3.1" % Test
).map(_.exclude("org.slf4j", "slf4j-jdk14"))
.map(_.excludeAll(ExclusionRule(organization = "ch.qos.logback")))
.map(_.excludeAll(ExclusionRule(organization = "QOS.ch")))
.map(_.excludeAll(ExclusionRule(artifact = "logback*")))
.map(_.excludeAll(ExclusionRule(artifact = "logback-classic")))
.map(_.exclude("ch.qos.logback", "logback-parent"))
.map(_.exclude("ch.qos.logback", "logback-core"))
.map(_.exclude("QOS.ch", "logback-parent"))
.map(_.exclude("", "logback-classic"))
}
It's not in the dependency tree for some reason:
$ activator "inspect tree test" |grep -i qos |wc -l
0
$ activator "inspect tree test" |grep -i logback |wc -l
0
Yet, when I run the test, it shows up!
$ activator test
[...]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/X/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.1.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/X/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
I'm at my wits' end. Help.

I have found that chaining the exclusions to the addition of libraryDependencies does not work, i.e.
libraryDependencies ++= {
val phantomVersion = "1.5.0"
Seq(
"net.jpountz.lz4" % "lz4" % "1.3.0",
"org.xerial.snappy" % "snappy-java" % "1.1.1.6",
"com.websudos" %% "phantom-dsl" % phantomVersion,
"com.websudos" %% "phantom-testing" % phantomVersion % Test,
"org.scalatestplus" %% "play" % "1.2.0" % Test,
"org.cassandraunit" % "cassandra-unit" % "2.1.3.1" % Test
).map(_.exclude("org.slf4j", "slf4j-jdk14"))
.map(_.excludeAll(ExclusionRule(organization = "ch.qos.logback")))
.map(_.excludeAll(ExclusionRule(organization = "QOS.ch")))
.map(_.excludeAll(ExclusionRule(artifact = "logback*")))
.map(_.excludeAll(ExclusionRule(artifact = "logback-classic")))
.map(_.exclude("ch.qos.logback", "logback-parent"))
.map(_.exclude("ch.qos.logback", "logback-core"))
.map(_.exclude("QOS.ch", "logback-parent"))
.map(_.exclude("", "logback-classic"))
}
Instead, add the exclusions using the undocumented SettingKey.~= function (http://www.scala-sbt.org/0.13.12/api/index.html#sbt.SettingKey) on the following line after adding new dependencies, i.e.
libraryDependencies ++= {
val phantomVersion = "1.5.0"
Seq(
"net.jpountz.lz4" % "lz4" % "1.3.0",
"org.xerial.snappy" % "snappy-java" % "1.1.1.6",
"com.websudos" %% "phantom-dsl" % phantomVersion,
"com.websudos" %% "phantom-testing" % phantomVersion % Test,
"org.scalatestplus" %% "play" % "1.2.0" % Test,
"org.cassandraunit" % "cassandra-unit" % "2.1.3.1" % Test
)
libraryDependencies ~= { _.map(_.exclude("org.slf4j", "slf4j-jdk14"))
.map(_.excludeAll(ExclusionRule(organization = "ch.qos.logback")))
.map(_.excludeAll(ExclusionRule(organization = "QOS.ch")))
.map(_.excludeAll(ExclusionRule(artifact = "logback*")))
.map(_.excludeAll(ExclusionRule(artifact = "logback-classic")))
.map(_.exclude("ch.qos.logback", "logback-parent"))
.map(_.exclude("ch.qos.logback", "logback-core"))
.map(_.exclude("QOS.ch", "logback-parent"))
.map(_.exclude("", "logback-classic"))
}
I don't know why this gives different behaviour, but with Play Framework 2.4.3 and SBT 0.13.8 this successfully excludes logback-classic and slf4j.
Note that you can chain exclude and excludeAll method calls to avoid repeatedly calling map so your code can be simplified to:
libraryDependencies ++= {
val phantomVersion = "1.5.0"
Seq(
"net.jpountz.lz4" % "lz4" % "1.3.0",
"org.xerial.snappy" % "snappy-java" % "1.1.1.6",
"com.websudos" %% "phantom-dsl" % phantomVersion,
"com.websudos" %% "phantom-testing" % phantomVersion % Test,
"org.scalatestplus" %% "play" % "1.2.0" % Test,
"org.cassandraunit" % "cassandra-unit" % "2.1.3.1" % Test
)
libraryDependencies ~= { _.map(_
.exclude("org.slf4j", "slf4j-jdk14"))
.exclude("ch.qos.logback", "logback-classic"))
}
Edit: After further investigation, I believe that this is required because the libraryDependencies already contain logback-classic from the Play plugin prior to parsing the build.sbt file. You can exclude the libraries in plugin.sbt, but if you are using the PlayScala plugin by enablePlugins(PlayScala)(https://www.playframework.com/documentation/2.5.x/NewApplication) this cannot be excluded so you have to add the exclusions separately.

There is a better way to do this. First you should identify which dependency is causing the problem in the first place.
Add this to plugins.sbt:
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.5")
Now you have a new command available in sbt, namely whatDependsOn. Using this, you can trial:
whatDependsOn ch.qos.logback logback-classic 1.1.1
Then I would suggest removing the dependency from the exact place, not mapping the entire configuration:
libraryDependencies ++= {
val phantomVersion = "1.5.0"
Seq(
"net.jpountz.lz4" % "lz4" % "1.3.0",
"org.xerial.snappy" % "snappy-java" % "1.1.1.6",
"com.websudos" %% "phantom-dsl" % phantomVersion excludeAll(
ExclusionRule(...)
),
"com.websudos" %% "phantom-testing" % phantomVersion % Test,
"org.scalatestplus" %% "play" % "1.2.0" % Test,
"org.cassandraunit" % "cassandra-unit" % "2.1.3.1" % Test
)
)

Related

How to patch scala compiler classes in an sbt project?

I am stumped. It should be a non-issue.
I took a patch from Scala 2 repository for a bug which crashes the compiler, compiled those classes separately against "scala-lang" % "scala-compiler" % "2.13.10" and packed into a jar. Then I tried to put them on boot class path of SBT, so they would be picked over the class loader used by the compiler:
scalacOptions ++= Seq(
"-JXbootclasspath/a:/home/turin/scala/patches/ConcurrentModificationExceptionPatch.jar",
"-bootclasspath:/home/turin/scala/patches/ConcurrentModificationExceptionPatch.jar",
)
The options above didn't work, and neither did
javaOptions ++= Seq(
"-Xbootclasspath/a:/home/turin/scala/patches/ConcurrentModificationExceptionPatch.jar",
)
I know the compiler does not see them, because line numbers in the stack trace of the exception thrown during compilation match release files, not 'my' files.
SBT recognizes the Scalac bootclasspath, but, apparently, not the java one. Here's the output generated with "-Ylog-classpath":
scalac: Classpath built from (-bootclasspath = /home/turin/scala/patches/ConcurrentModificationExceptionPatch.jar -encoding = UTF-8 -feature = true -g = vars -verbose = true -Wconf = List(msg=export:silent, cat=other-match-analysis&msg=Singleton():silent, cat=deprecation&msg=foldLeft:silent, cat=deprecation&msg=foldRight:silent, cat=deprecation:w, cat=feature:w, cat=feature:w, cat=deprecation:ws, cat=feature:ws, cat=optimizer:ws) -Wperformance = PerformanceWarnings.ValueSet() -Wunused = UnusedWarnings.ValueSet(patvars, privates, locals) -Xlint = LintWarnings.ValueSet(nullary-unit, poly-implicit-overload, option-implicit, delayedinit-select, implicit-not-found) -classpath = /home/turin/porn/oldsql/target/scala-2.13/classes:/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/java-diff-utils/java-diff-utils/4.12/java-diff-utils-4.12.jar:/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/net/bytebuddy/byte-buddy/1.12.23/byte-buddy-1.12.23.jar:/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.9.0/jna-5.9.0.jar:/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline/3.21.0/jline-3.21.0.jar:/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.13.10/scala-compiler-2.13.10.jar:/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.10/scala-library-2.13.10.jar:/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.10/scala-reflect-2.13.10.jar -language = languageFeatures.ValueSet(existentials, higherKinds, implicitConversions, reflectiveCalls) -Vlog = List(0-100) -Vclasspath = true -Vprint-types = true -Vimplicits = true -Vimplicits-verbose-tree = true -Vreflective-calls = true)
Sorry for the mile long line, I didn't want to edit it for fear of removing something important.
scalac: Defaults: object Defaults {
scalaHome =
javaBootClassPath =
scalaLibDirFound = None
scalaLibFound =
scalaBootClassPath =
scalaPluginPath = misc/scala-devel/plugins
}
calac: Calculated: object Calculated {
scalaHome =
javaBootClassPath =
javaExtDirs =
javaUserClassPath =
useJavaClassPath = false
scalaBootClassPath = /home/turin/scala/patches/ConcurrentModificationExceptionPatch.jar
scalaExtDirs =
userClassPath =
/home/turin/porn/oldsql/target/scala-2.13/classes
/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/java-diff-utils/java-diff-utils/4.12/java-diff-utils-4.12.jar
/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/net/bytebuddy/byte-buddy/1.12.23/byte-buddy-1.12.23.jar
/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.9.0/jna-5.9.0.jar
/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline/3.21.0/jline-3.21.0.jar
/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.13.10/scala-compiler-2.13.10.jar
/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.10/scala-library-2.13.10.jar
/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.10/scala-reflect-2.13.10.jar
sourcePath =
}
scalac: After java boot/extdirs classpath has 9 entries:
ZipArchiveClassPath(/home/turin/scala/patches/ConcurrentModificationExceptionPatch.jar,None)
DirectoryClassPath(/home/turin/porn/oldsql/target/scala-2.13/classes)
ZipArchiveClassPath(/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/io/github/java-diff-utils/java-diff-utils/4.12/java-diff-utils-4.12.jar,None)
ZipArchiveClassPath(/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/net/bytebuddy/byte-buddy/1.12.23/byte-buddy-1.12.23.jar,None)
ZipArchiveClassPath(/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.9.0/jna-5.9.0.jar,None)
ZipArchiveClassPath(/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline/3.21.0/jline-3.21.0.jar,None)
ZipArchiveClassPath(/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.13.10/scala-compiler-2.13.10.jar,None)
ZipArchiveClassPath(/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.10/scala-library-2.13.10.jar,None)
ZipArchiveClassPath(/home/turin/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.10/scala-reflect-2.13.10.jar,None)
What am I doing wrong?

Understanding pyinstaller

I write a few small scripts to python ever so often, but sometimes Pyinstaller does not compile my code right, and i dont have a deep enough understanding to know why.
So i am so sorry that i am posting my whole (i promise) short code, but i have NO idea where the problem could be. The code works fine when its run in pyCharm, so the focus is on why Pyinstaller dosent not compile it right.
When i run Pyinstaller i get no errors, but when i then run the exe i get the following: Current thread 0x000027d0 (most recent call first):
So if anyone could have a look through and give me direction to where i go wrong that would be great. Thanks in advance.
from math import *
from tkinter import *
def calc_outcome(k,N,n,r):
a = factorial(k - 1) / (factorial(r - 1) * factorial(k - r))
b = factorial(N - k) / (factorial(n - r) * factorial(N - k - n + r))
c = factorial(N) / (factorial(N - n) * factorial(n))
return(a * b)/c
def runner():
N = int(PopulationSize.get())
n = int(SampleSize.get())
startHand = 7
avgGameLenght = 15
print("Draw:")
string = " "
for x in range(1,n+1):
string += '{:>10}'.format(str(x) + "/" + str(n))
print(string + "\n")
holder = []
for x in range(N):
holder.append(0)
for k in range(N+1):
string = ""
string += str('{:2}'.format(k)) + ": "
if k == startHand+1 or k == avgGameLenght+1:
print("-"*((n*10)+4))
for r in range(1,n+1):
try:
holder[r-1] += calc_outcome(k,N,n,r)
string += str('{:>10}'.format('{:.1%}'.format(holder[r - 1])))
except:
string += '{:>10}'.format("-")
if k > 0:
print(string)
def changeEntry(PS, PM):
if PS == "P":
if PM == "P":
holder = int(PopulationSize.get()) + 1
PopulationSize.delete(0,END)
PopulationSize.insert(0,holder)
else:
holder = int(PopulationSize.get()) -1
if holder < 0: holder = 0
PopulationSize.delete(0, END)
PopulationSize.insert(0, holder)
else:
if PM == "P":
holder = int(SampleSize.get()) + 1
SampleSize.delete(0,END)
SampleSize.insert(0,holder)
else:
holder = int(SampleSize.get()) -1
if holder < 0: holder = 0
SampleSize.delete(0, END)
SampleSize.insert(0, holder)
vindue = Tk()
PopulationSize = Entry(vindue, width=3, font="Arial, 24", justify=CENTER)
SampleSize = Entry(vindue, width=3, font="Arial, 24", justify=CENTER)
PopulationSize.insert(0,60)
SampleSize.insert(0,4)
knap = Button(vindue, text="Run", command=runner, font="Arial, 16")
knapPP = Button(vindue, text="P+", font="Arial, 16", command=lambda: changeEntry("P","P"))
knapPM = Button(vindue, text="P-", font="Arial, 16", command=lambda: changeEntry("P","M"))
knapSP = Button(vindue, text="S+", font="Arial, 16", command=lambda: changeEntry("S","P"))
knapSM = Button(vindue, text="S-", font="Arial, 16", command=lambda: changeEntry("S","M"))
PopulationSize.grid(row=0, column=0)
SampleSize.grid(row=1, column=0)
knap.grid(row=2, column=0)
knapPP.grid(row=0, column=1)
knapPM.grid(row=0, column=2)
knapSP.grid(row=1, column=1)
knapSM.grid(row=1, column=2)
vindue.mainloop()
This is the output from Pyinstaller (just in case its needed):
C:\..\Scripts>pyinstaller.exe --onefile C:\..\Test.py
394 INFO: PyInstaller: 3.4
395 INFO: Python: 3.4.4
396 INFO: Platform: Windows-10-10.0.17134
401 INFO: wrote C:\..\Test.spec
404 INFO: UPX is not available.
405 INFO: Extending PYTHONPATH with paths
['C:\\..', 'C:\\..\\Scripts']
405 INFO: checking Analysis
446 INFO: checking PYZ
473 INFO: checking PKG
480 INFO: Building because C:\..\Test.exe.manifest changed
480 INFO: Building PKG (CArchive) PKG-00.pkg
2302 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
2305 INFO: Bootloader c:\..\run.exe
2306 INFO: checking EXE
2312 INFO: Building because manifest changed
2312 INFO: Building EXE from EXE-00.toc
2314 INFO: Appending archive to EXE C:\..\Test.exe
2711 INFO: Building EXE from EXE-00.toc completed successfully.
C:\..\Scripts>

Running R in Scala using rscala causing issues

I recently came across an R package called rscala. https://darrenjw.wordpress.com/tag/rscala/
I tried executing the example however the program never completed running. I am not sure what may be wrong. Whenever I try to instantiate RClient, it seems to run forever. Please help.
For me the following code runs:
import breeze.stats.distributions._
import breeze.linalg._
import org.ddahl.rscala.RClient
object ScalaToRTest {
def main(args: Array[String]): Unit = {
// first simulate some data consistent with a Poisson regression model
val x = Uniform(50,60).sample(1000)
val eta = x map { xi => (xi * 0.1) - 3 }
val mu = eta map { math.exp }
val y = mu map { Poisson(_).draw }
// call to R to fit the Poission regression model
val R = RClient() // initialise an R interpreter
R.x=x.toArray // send x to R
R.y=y.toArray // send y to R
R.eval("mod <- glm(y~x,family=poisson())") // fit the model in R
// pull the fitted coefficents back into scala
val beta = DenseVector[Double](R.evalD1("mod$coefficients"))
// print the fitted coefficents
println(beta)
}
}
Output:
DenseVector(-3.1683714618415855, 0.1031332817387318)
build.sbt
name := "scalaRdemo"
version := "0.1"
scalaVersion := "2.12.3"
scalacOptions ++= Seq("-unchecked", "-deprecation", "-feature")
resolvers ++= Seq(
"Sonatype Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots/",
"Sonatype Releases" at "https://oss.sonatype.org/content/repositories/releases/"
)
libraryDependencies += "org.scalanlp" %% "breeze-natives" % "0.13.2"
libraryDependencies += "org.scalanlp" %% "breeze" % "0.13.2"
libraryDependencies += "org.ddahl" %% "rscala" % "2.3.5"

SBT run task after all subprojects

I want to write a task that is run after all subproject tasks are complete.
Eg, if I do
sbt a b then after task a is complete on ALL subprojects I want to execute task b. I don't want to do (a b) on each project.
Is that possible?
In fact, I'll take it so that I modify the build.sbt directly. I don't necessarily have to specify it at the command line.
I wrote a blog post on the subject: sequencing tasks with sbt-sequential.
addCommandAlias
Here's an example. We'll define a custom task a in sub1 and sub2 and b in root. The simplest way to achieve sequential execution is using addCommandAlias, so we'll just do that.
lazy val a = taskKey[Unit]("a")
lazy val b = taskKey[Unit]("b")
lazy val root = (project in file(".")).
aggregate(sub1, sub2).
settings(addCommandAlias("ab", ";a;b"): _*).
settings(
b := {
println("b")
}
)
lazy val sub1 = (project in file("sub1")).
settings(a := println("a - sub1"))
lazy val sub2 = (project in file("sub2")).
settings(a := println("a - sub2"))
You can run this from shell as sbt ab.
$ sbt ab
[info] Loading global plugins from ...
[info] Loading project definition from ...
[info] Set current project to root (in build ...)
a - sub2
a - sub1
[success] Total time: 0 s, completed Nov 22, 2014 8:36:18 PM
b
[success] Total time: 0 s, completed Nov 22, 2014 8:36:18 PM
Def.taskDyn
Here's another example. This time using Def.taskDyn, which is also featured in the blog post.
I'm constructing a ScopeFilter from the aggregate and then I'm dispatching task a to them.
lazy val a = taskKey[File]("a")
lazy val b = taskKey[Seq[File]]("b")
lazy val root = (project in file(".")).
aggregate(sub1, sub2).
settings(
b := (Def.taskDyn {
val proj = thisProject.value
val filter = ScopeFilter(inProjects(proj.aggregate: _*))
Def.task {
val allValues = a.all(filter).value
println(allValues.mkString(","))
allValues
}
}).value
)
lazy val sub1 = (project in file("sub1")).
settings(a := new File("a"))
lazy val sub2 = (project in file("sub2")).
settings(a := new File("b"))
You can run this from shell as sbt b.
$ sbt b
[info] Loading global plugins from ...
[info] Loading project definition from ...
[info] Set current project to root (in build ...)
a,b
[success] Total time: 0 s, completed Nov 23, 2014 9:42:16 PM

Which of these is pythonic? and Pythonic vs. Speed

I'm new to python and just wrote this module level function:
def _interval(patt):
""" Converts a string pattern of the form '1y 42d 14h56m'
to a timedelta object.
y - years (365 days), M - months (30 days), w - weeks, d - days,
h - hours, m - minutes, s - seconds"""
m = _re.findall(r'([+-]?\d*(?:\.\d+)?)([yMwdhms])', patt)
args = {'weeks': 0.0,
'days': 0.0,
'hours': 0.0,
'minutes': 0.0,
'seconds': 0.0}
for (n,q) in m:
if q=='y':
args['days'] += float(n)*365
elif q=='M':
args['days'] += float(n)*30
elif q=='w':
args['weeks'] += float(n)
elif q=='d':
args['days'] += float(n)
elif q=='h':
args['hours'] += float(n)
elif q=='m':
args['minutes'] += float(n)
elif q=='s':
args['seconds'] += float(n)
return _dt.timedelta(**args)
My issue is with the for loop here i.e the long if elif block, and was wondering if there is a more pythonic way of doing it.
So I re-wrote the function as:
def _interval2(patt):
m = _re.findall(r'([+-]?\d*(?:\.\d+)?)([yMwdhms])', patt)
args = {'weeks': 0.0,
'days': 0.0,
'hours': 0.0,
'minutes': 0.0,
'seconds': 0.0}
argsmap = {'y': ('days', lambda x: float(x)*365),
'M': ('days', lambda x: float(x)*30),
'w': ('weeks', lambda x: float(x)),
'd': ('days', lambda x: float(x)),
'h': ('hours', lambda x: float(x)),
'm': ('minutes', lambda x: float(x)),
's': ('seconds', lambda x: float(x))}
for (n,q) in m:
args[argsmap[q][0]] += argsmap[q][1](n)
return _dt.timedelta(**args)
I tested the execution times of both the codes using timeit module and found that the second one took about 5-6 seconds longer (for the default number of repeats).
So my question is:
1. Which code is considered more pythonic?
2. Is there still a more pythonic was of writing this function?
3. What about the trade-offs between pythonicity and other aspects (like speed in this case) of programming?
p.s. I kinda have an OCD for elegant code.
EDITED _interval2 after seeing this answer:
argsmap = {'y': ('days', 365),
'M': ('days', 30),
'w': ('weeks', 1),
'd': ('days', 1),
'h': ('hours', 1),
'm': ('minutes', 1),
's': ('seconds', 1)}
for (n,q) in m:
args[argsmap[q][0]] += float(n)*argsmap[q][1]
You seem to create a lot of lambdas every time you parse. You really don't need a lambda, just a multiplier. Try this:
def _factor_for(what):
if what == 'y': return 365
elif what == 'M': return 30
elif what in ('w', 'd', 'h', 's', 'm'): return 1
else raise ValueError("Invalid specifier %r" % what)
for (n,q) in m:
args[argsmap[q][0]] += _factor_for([q][1]) * n
Don't make _factor_for a method's local function or a method, though, to speed things up.
(I have not timed this, but) if you're going to use this function often it might be worth pre-compiling the regex expression.
Here's my take on your function:
re_timestr = re.compile("""
((?P<years>\d+)y)?\s*
((?P<months>\d+)M)?\s*
((?P<weeks>\d+)w)?\s*
((?P<days>\d+)d)?\s*
((?P<hours>\d+)h)?\s*
((?P<minutes>\d+)m)?\s*
((?P<seconds>\d+)s)?
""", re.VERBOSE)
def interval3(patt):
p = {}
match = re_timestr.match(patt)
if not match:
raise ValueError("invalid pattern : %s" % (patt))
for k,v in match.groupdict("0").iteritems():
p[k] = int(v) # cast string to int
p["days"] += p.pop("years") * 365 # convert years to days
p["days"] += p.pop("months") * 30 # convert months to days
return datetime.timedelta(**p)
update
From this question, it looks like precompiling regex patterns does not bring about noticeable performance improvement since Python caches and reuses them anyway. You only save the time it takes to check the cache which, unless you are repeating it numerous times, is negligible.
update2
As you quite rightly pointed out, this solution does not support interval3("1h 30s" + "2h 10m"). However, timedelta supports arithmetic operations which means one you can still express it as interval3("1h 30s") + interval3("2h 10m").
Also, as mentioned by some of the comments on the question, you may want to avoid supporting "years" and "months" in the inputs. There's a reason why timedelta does not support those arguments; it cannot be handled correctly (and incorrect code are almost never elegant).
Here's another version, this time with support for float, negative values, and some error checking.
re_timestr = re.compile("""
^\s*
((?P<weeks>[+-]?\d+(\.\d*)?)w)?\s*
((?P<days>[+-]?\d+(\.\d*)?)d)?\s*
((?P<hours>[+-]?\d+(\.\d*)?)h)?\s*
((?P<minutes>[+-]?\d+(\.\d*)?)m)?\s*
((?P<seconds>[+-]?\d+(\.\d*)?)s)?\s*
$
""", re.VERBOSE)
def interval4(patt):
p = {}
match = re_timestr.match(patt)
if not match:
raise ValueError("invalid pattern : %s" % (patt))
for k,v in match.groupdict("0").iteritems():
p[k] = float(v) # cast string to int
return datetime.timedelta(**p)
Example use cases:
>>> print interval4("1w 2d 3h4m") # basic use
9 days, 3:04:00
>>> print interval4("1w") - interval4("2d 3h 4m") # timedelta arithmetic
4 days, 20:56:00
>>> print interval4("0.3w -2.d +1.01h") # +ve and -ve floats
3:24:36
>>> print interval4("0.3x") # reject invalid input
Traceback (most recent call last):
File "date.py", line 19, in interval4
raise ValueError("invalid pattern : %s" % (patt))
ValueError: invalid pattern : 0.3x
>>> print interval4("1h 2w") # order matters
Traceback (most recent call last):
File "date.py", line 19, in interval4
raise ValueError("invalid pattern : %s" % (patt))
ValueError: invalid pattern : 1h 2w
Yes, there is. Use time.strptime instead:
Parse a string representing a time
according to a format. The return
value is a struct_time as returned
by gmtime() or localtime().

Resources