we use a tool that operates on a certain database updating certain values when we makes some changes to the tool.
This takes a long time for some simple task.
I just need to find out which table,column and the value for that column gets updated.
for this i need to find out the whole database which column_name has value "XYZ" and the corresponding table/tables.
Any scripts for these.
Just because something can be done does not mean it should be done.
I know you've got your process designed this way and you very likely don't want to change it but, really, your life will get a whole lot better if you redesign this to avoid doing something that really, seriously, shouldn't be done. Searching through every text field in an entire database in search of some magical character string is a Bad Idea. It's actually only ONE of The Big Bad Ideas and it probably isn't the Baddest Idea, but it's a big enough bad enough idea that you should give Serious Consideration to doing something else, better.
OK, so what's wrong with it?
First, it indicates that you're not using a database, you're using a midden. You dump stuff in and then hope to dig it out later. This is the kind of thing people did thousands of years ago (it was popular back when flint was cutting edge technology), and while it helps keep archaeologists employed digging through these trash heaps, we are software developers, not archaeologists, and we don't want to have to do this kind of thing on a regular basis.
Second, this is a serious performance killer. You're going to either write some god-awful static code to laboriously check every field in every table, or you're going to write some middling-bright code to dynamically create some even more god-awful query that will laboriously check every field in every table. The word to focus on here is "laborious". And "god-awful", if it comes to that. Scanning through every row in every table in your database and testing every field in all of those every rows is going to be slow. Very, very slow. It's going to be dead-turtle-on-the-side-of-the-road-with-tire-marks-on-its-shell slow. This is not a good thing to do, unless you own stock in the local electric utility and want to make sure every generated electron has a happy home in your employers computer.
Third, people will have strong emotions when they see your code. Those destined for careers in management will laugh, for they know that they won't have to maintain it or try to solve the performance issues. The technically challenged will cry, because they'll know there's nothing they can do to fix it. The true Code Warriors will stare in amazement for a moment, and will then grit their teeth, hunt you down, and beat you to death with their ceremonial Wands Of Green-Bar, for only they will know that this evil could have been prevented.
So give some thought to a re-design. Once again, just because something can be done does not mean it should be done.
Share and enjoy.
Related
There seems to competing best practices for user stories that I have not been able to come up with a way to deal with. Mainly:
Keep user stories small.
User stories should deliver customer values and therefore engineering focused item such as refactoring should not be separate user story and should be part of on-going work.
I feel like it's hard to achieve both of these at the same time.
For example, sometimes you need to refactor things. Some code is complex and just takes time. If it's done as part of the feature user story, then that user story will get bigger. But the refactor should not be a user story since it doesn't deliver customer values. You could argue that maybe we shouldn't let the code got checked-in in the first place, but requirement changes and therefore assumption changes so I don't think that's realistic expectation.
Another example could be dealing with starting a new project; we need to setup the repository, the project, the CI/CD pipelines. All of these are infrastructure work items and does not deliver any direct customer values. I guess in this case, we could use "engineer" as the customer but there are some debate out there whether that's a good practice or not.
Now I could bend the rule and have these as user stories. But I am curious if people are following the scrum rule strictly, is there tips and tricks to achieve both of these requirements?
Scrum does not prescribe any format for the Product Backlog items. So your Product Backlog can contain user stories but also any other kind of items.
As you are saying user stories should deliver value, typically meeting the INVEST acronym but your Product Backlog can contain items related to make functionality possible, such as upgrading libraries, implementing a log system or whatever help the team to potentially release the product. Taking into account that in your Product Backlog there will be bugs too, so it´s not just about new functionalities. So don´t fool yourself making up users who don´t exist.
Being saying that, usually a big refactor is a smell of a problem not tackled at the proper time (generating a high technical debt, a weak DoD and so on). That´s the reason many authors say that you should refactor at the same time your are developing new functionalities (your code should be in good form).
Finally, if you are in such a scenario that you need a big refactor, my advice is to try to decompose the refactor in small pieces and create a good harness of test to refactor with confidence.
I can think of several ways you can cut stories. However, keep in mind that these are ideas that can work but can also make no sense depending on the work. I only want to share my experience.
think about the three basic areas: User Interface, Data Calculation, Data Retrieval, try to cut the story along these lines
if you have to do repetitions, create a story for each iteration
try to do the first implementation as basic as possible, then create more stories for upgrading and fine tuning
even if a story seems to be straightforward, try to cut it; it is amazing how much better you can discuss as a team about smaller items and how much better you can refine it, and you it forces you to think about the details and this really can make a difference
What is the preferred (or just a good) pattern for a multi-part Codecept.js scenario, such as this:
Select file to upload.
Clear selection.
Select file to upload after having cleared selection.
I can do this in a single scenario and use I.say to delineate the parts, but it feels like I should be able to write these as independent scenarios and use .only on part 2, for example, and have part 1 run prior to part 2, because it depends on it.
I would also want to skip parts 2 and 3 if part 1 fails when running the whole suite.
I like thinking about behaviour in terms of capabilities. I can see that you have a couple here:
Uploading files
Correcting mistakes while uploading files.
So I would expect these to be in two scenarios:
One where you actually upload the file
One where you correct mistakes you make.
A lot of people say there should only be one "When" in scenarios, but that doesn't take into account interactions with people (including your past mistaken self) or time. In this situation, it's the whole process of correcting the file upload that provides the value. I can't see any value in the intermediate steps, so I'd leave them all in one scenario.
If there's any different behaviour associated with different contexts (eg: you've already got too many files uploaded) or outcomes (eg: your file system doesn't have room) or rules (eg: your status means you qualify for super-fast upload) then I would expect those to be new scenarios. If you start getting to the point where there are a lot of scenarios associated with file uploads and different things that happen to them, that might be a good time to separate this scenario out. Right now I can't see any reason to do that.
Re failing the first part: if you're doing BDD right, you'll be talking through not just the behaviour of your system, but the behaviour of individual bits of code too. That should help produce a good design which minimizes the chances of having bugs. Really good BDD teams produce scenarios that hardly ever catch bugs!
The scenarios act as living documentation, rather than regression tests; helping future devs understand the value of the code and get it right, rather than nailing it down to stop them getting it wrong.
So I wouldn't worry about it failing. If it's doing that a lot, you've got a different problem. Code it as if it's going to be passing most of the time, and make sure it's readable and comprehensible. As long as you can see when it fails and work it out, even if that takes a little bit of time, it'll be fine.
Having said that, I'd be surprised if Codecept doesn't have at least an option to stop on failure. Most BDD tools don't continue a scenario after a failed step; it would be an odd thing to do.
As far as I know, you're not be able to set priority for execution in codeceptjs. Better make one test. it is also will be more flexible if you will need to add or delete some part.
EDIT: I've moved this question over to a more approprite StackExchange: https://crypto.stackexchange.com/questions/19150/ways-to-make-a-doomsday-cryptocurrency-which-becomes-untradable
Feel free to delete.
As a social experiment (not a money-making scheme), I'm interested in developing a crazy cryptocurrency which, by its very design, will become worthless and untradable after a certain point. Ideally, this would be a fixed date like "January 1st 2020," but it could also be after a predictable event occurs, such as after one million trades have been made, or once a super expensive computation completes. I, the designer of the currency, should not have any ability to prevent this "doomsday," postpone it, or even cause it to happen early. Lastly, my users should not have the ability to defy the doomsday by creating their own fork of the code.
The current idea I'm running with is to have there be a "super-key," which works as a universal private key and allows you to break into anyone's account. Since the accounts would cease to be secure, the currency would be worthless. I would generate this key, then encrypt it in such a way that it would take years (but not decades!) to crack, then release it publicly. There's a few downsides to this approach, mainly that I would not be able to guarantee that I myself don't own the super-key.
I'm wondering if anyone has any suggestions on how to implement this.
I'll include one other solution that I've found:
ftp://ftp.deas.harvard.edu/techreports/tr-22-06.pdf
This would rely upon a 3rd party service to gradually crack a file that would somehow break the currency (such as the aforementioned super-key).
so I need to maintain this old legacy project, where one part of it is amaturely written with Wordpess, lots of crappy custom plugins, lots of ducktaped scripts, that solve one or other problem, database is designed very very purely too, there is also other part written with Zend, which is thightly connected with WP part, there is also this other "masterpiece" project connected to first project data. Main table contains around 1.5M records and needs to be normalized too. now this big ball of nails "works", also it has lots of LOCs, which are result of bad foundations, so it is a huge pain to maintain. Thing is, the way I see this, by not rewriting we are loosing on the long term, because we lose flexibility both from technology and busi
ness perspective,plus it starts not to scale, but rewriting is a risk, plus we would need to convert old data to new data structures. Hacker part of me wants to break this, take a risk and do it right, but at the same time I am having a feeling that my immaturity tries to take a too big bite at once. So what do you think?
In situations like yours, 9 out of 10 times people suggest a rewrite and they are wrong.
Unless you have great application level knowledge about what the system is doing you will not be able to rewrite it successfully, quickly.
If the system is working today, but is crappy in many ways, and you have management buy-in (they own the software) to "fix stuffs", then I suggest an incremental approach will often be better than a full-on rewrite.
I suspect that the database is giving you the most headaches, so that may be the best place to start. Start by understanding the problem that it is currently solving and write that down. If there is no layer between the software and the db (other than jdbc or the like) add a layer. Once there is a layer separating the db from the application, it will be easier to change the db (and the layer) while minimizing the impact on the application.
At some point you will be happy with the first thing you changed. At that point, fix some other part. Repeat until the system is "more better".
Concerning risk: Taking risks is not bad, but being careless is terrible. Understand the risks and plan to mitigate them.
In situations like yours, 9 out of 10 times, I suggest to rewrite. Rationally, the situation a) won't get better, and b) will certainly get worse. You should bite the bullet before it's too late.
And by too late I mean something breaks completely and not only will you have to rewrite the whole thing, but your service will also be offline (ergo you may be also losing users/customers).
A good strategy in this situations is to "strangle" your application as described by Martin Fowler:
http://martinfowler.com/bliki/StranglerApplication.html
The strategy is to gradually create a new system around the edges of the old, letting it grow slowly over several years until the old system is strangled.
I already strangled a legacy application using this approach with great results and practically no offline time
This is more of a business-oriented programming question that I can't seem to figure out how to resolve. I work with a team of programmers who have been working with BASIC for over 20 years. I was brought in to help write the same software in .NET, only with updates and modern practices. The problem is that I can't seem to get any of the other 3 team members(all BASIC programmers, though one does .NET now as well) to understand how to correctly do a relational database. Here's the thing they won't understand:
We basically have a transaction that keeps track of a customer's tag information. We need to be able to track current transactions and past transactions. In the old system, a flat-file database was used that had one table that contained records with the basic current transaction of the customer, and another transaction that contained all the previous transactions of the customer along with important money information. To prevent redundancy, they would overwrite the current transaction with the history transactions-(the history file was updated first, then the current one.) It's totally unneccessary since you only need one transaction table, but my supervisor or any of my other two co-workers can't seem to understand this. How exactly can I convince them to see the light so that we won't have to do ridiculous amounts of work and end up hitting the datatabse too many times? Thanks for the input!
Firstly I must admit it's not absolutely clear to me from your description what the data structures and logic flows in the existing structures actually are. This does imply to me that perhaps you are not making yourself clear to your co-workers either, so one of your priorities must be to be able explain, either verbally or preferably in writing and diagrams, the current situation and the proposed replacement. Please take this as an observation rather than any criticism of your question.
Secondly I do find it quite remarkable that programmers of 20 years experience do not understand relational databases and transactions. Flat file coding went out of the mainstream a very long time ago - I first handled relational databases in a commercial setting back in 1988 and they were pretty commonplace by the mid-90s. What sector and product type are you working on? It sounds possible to me that you might be dealing with some sort of embedded or otherwise 'unusual' system, in which case you do need to make sure that you don't have some sort of communication issue and you're overlooking a large elephant that hasn't been pointed out to you - you wouldn't be the first 'consultant' brought into a team who has been set up in some manner by not being fed the appropriate information. That said such archaic shops do still exist - one of my current clients systems interfaces to a flat-file based system coded in COBOL, and yes, it is hell to manage ;-)
Finally, if you are completely sure of your ground and you are faced with a team who won't take on board your recommendations - and demonstration code is a good idea if you can spare the time -then you'll probably have to accept the decision gracefully and move one. Myself in this position I would attempt to abstract out the issue - can the database updates be moved into stored procedures for example so the code to update both tables is in the SP and can be modified at a later date to move to your schema without a corresponding application change? Make sure your arguments are well documented and recorded so you can revisit them later should the opportunity arise.
You will not be the first coder who's had to implement a sub-optimal solution because of office politics - use it as a learning experience for your own personal development about handling such situations and commiserate yourself with the thought you'll get paid for the additional work. Often the deciding factor in such arguments is not the logic, but the 'weight of reputation' you yourself bring to the table - it sounds like having been brought in you don't have much of that sort of leverage with your team, so you may have to work on gaining a reputation by exceling at implementing what they do agree to do before you have sufficient reputation in subsequent cases - you need to be modded up first!
Sometimes you can't.
If you read some XP books, they often say that one of your biggest hurdles will be convincing your team to abandon what they have always done.
Generally they will recommend letting people who can't adapt go to other projects (Or just letting them go).
Code reviews might help in your case. Mandatory code reviews of every line of code is not unheard of.
Sometime the best argument is an example. I'd write a prototype (or a replacement if not too much work). With an example to examine it will be easier to see the pros and cons of a relational database.
As an aside, flat-file databases have their places since they are so much easier to "administer" than a true relational database. Keep an open mind. ;-)
I think you may have to lead by example - when people see that the "new" way is less work they will adopt it (as long as you don't rub their noses in it).
I would also ask yourself whether the old design is actually causing a problem or whether it is just aesthetically annoying. It's important to pick your battles - if the old design isn't causing a performance problem or making the system hard to maintain you may want to leave the old design alone.
Finally, if you do leave the old design in place, try and abstract the interface between your new code and the old database so if you do persuade your co-workers to improve the design later you can drop the new schema in without having to change anything else.
It is difficult to extract a whole lot except general frustration from the original question.
Yes, there are a lot of techniques and habits long-timers pick up over time that can be useless and even costly in light of technology changes. Some things that made sense when processing power, memory, and even disk was expensive can be foolish attempts at optimization now. It is also very much the case that people accumulate bad habits and bad programming patterns over time.
You have to be careful though.
Sometimes there are good reasons for the things those old timers do. Sadly, they may not even be able to verbalize the "why" - if they even know why anymore.
I see a lot of this sort of frustration when newbies come into an enterprise software development shop. It can be bad even when the environment is all fairly modern technology and tools. If most of your experience is in writing small-community desktop and Web applications a lot of what you "know" may be wrong.
Often there are requirements for transaction journaling at a level above what your DBMS may do. Quite often it can be necessary to go beyond DB transaction semantics in order to ensure time-sequence correctness, once and only once updating, resiliancy, and non-repudiation.
And this doesn't even begin to address the issues involved in enterprise or inter-enterprise scalability. When you begin to approach half a million complex transactions a day you will find that RDBMS technology fails you. Because relational databases are not designed to handle high transaction volumes you must often break with standard paradigms for normalization and updating. Conventional RDBMS locking techniques can destroy scalability no matter how much hardware you throw at the problem.
It is easy to dismiss all of it as stodginess or general wrong-headedness - even incompetence. But be careful because this isn't always the case.
And by the way: There are other models besides the RDBMS, and the alternative to an RDBMS is not necessarily "flat files" - contrary to the experience of of most coders today. There are transactional hierarchical DBMSs that can handle much higher throughput than an RDBMS. IMS is still very much alive in large IBM shops, for example. Other vendors offer similar software for different platforms.
Of course in a 4-man shop maybe none of this applies.
Sign them up for some decent trainings and then it's up to you to convince them that with new technologies a lot more is possible (or at least easier!).
But I think the most important thing here is that professional, certified trainers teach them the basics first. They will be more impressed by that instead of just one of their colleagues telling them: "hey, why not use this?"
Related post here.
The following may not apply in yr situation, but you make very little mention of technical details, so I thought I'd mention it...
Sometimes, if the access patterns are very different for current data than for historical data (I'm making this example up, but say that Current data is accessed 1000s of times per second, and accesses a small subset of columns, and all current data fits in less than 1 GB, whereas, say, historical data uses 1000s of GBs, is accessed only 100s of times per day, and access is to all columns),
then, what your co-workers are doing would make perfect sense, for performance optimization. By separating the current data (albiet redundantly) you can optimize the indices and data structures in that table, for the higher frequency access paterns that you could not do in the historical table.
Not everything that is "academically", or "technically" correct from a purely relational perspective makes sense when applied in an actual practical situation.