Stopping a margin transition at its initial point - css

Question

Alright, here's what I have a nifty little scroller (they've been done to death but I wanted to take my hand at making one) and I want it to scroll to the left for X amount of pixels, which is no problem. The problem is, if the user hovers over the left side and starts scrolling the text back, I want it to stop at its initial starting point.
body {
background-color: black;
}
.wrapper {
position: absolute;
left: 50%;
margin-left: -250px;
overflow: hidden;
width: 500px;
height: 300px;
z-index: 98;
border: 3px solid 2a3439;
background-color: #353839;
}
.scroller {
color: white;
width: 1980px;
background-color: transparent;
height: 100%;
position: relative;
padding: 5px;
margin-left: 0px;
transition: all 2s linear 10s;
-moz-transition: all 2s linear 10s;
-webkit-transition: all 2s linear 10s;
}
.go-left:hover~.scroller {
margin-left: -250px;
transition: all 2s linear;
-moz-transition: all 2s linear;
-webkit-transition: all 2s linear ;
}
.go-left {
opacity: 0;
height: 300px;
background-color: red;
width: 200px;
z-index: 99;
position: absolute;
top: 0;
}
.go-right {
opacity: 0;
height: 300px;
background-color: red;
width: 200px;
z-index: 99;
position: absolute;
top: 0;
left: 300px
}
.go-right:hover+.scroller {
margin-left: 250px;
transition: all 2s linear;
-moz-transition: all 2s linear;
-webkit-transition: all 2s linear;
}
.image {
display: block;
overflow: hidden;
Height: 490px;
}
In that example, if you hover to the left I don't want it to move past it's starting point, so clamping it. You can see the working model on jsfiddle https://jsfiddle.net/Kaanha/puy6t4og/51/
This wouldn't be hard using some java, but the main thing here is, I need it to be pure CSS only. No java.
Thanks in advance for all help!