Neighbours preveting me from fencign my property over passage

I have a property of 3 acres in Kerala. We have 4 neighbors on one side of the property. They had been using the border of our property to walk to their homes from a panchayat road. About 10 years ago, the neighbors came to us and asked us to give 3ft of our land along the property border so that they can use it to bring vehicles to their land. They had also agreed to take 3ft of their own property as part of the road. We agreed and built a temporary fence on that side after leaving 3ft distance. They did the same. However the land was never given to anyone and legally is still part of our property. A couple of years ago the neighbors reclaimed the 3ft they had given for the road and made it part of their property. When I asked them about it, they got angry at me. Then they started asking me to officially give away the 3ft land fr4om my property to the panchayat. I rejected. Last week I decided to replace the temporary fence with a permanent one keeping the 3ft land out side the fence as it has been for years. However the neighbors did not let the construction workers enter the land and is not letting me build the fence. Now they are saying that I must given them 4ft of land and should officially give it to the Panchayat. Unfortunately I don't live in the same Panchayat the property in question is in. The Panchayat officials are against me and is demanding that I give away the land. From a legal stand point, should I give away the land? What are my options?