Since it is your grandfather's property and your mother is one of the legal heirs to your deceased grandfather, upon his intestate death she is entitled to to her legitimate share in the property as a right in the capacity of successor in interest.
Though she got married in the year 1955, since her father expired in the year 1979, she is entitled to a rightful share as per Hindu Succession act, 1956.
She can file a partition suit and also can seek a share in the mesne profits so far enjoyed by her brother and also entitled to have a detailed account of the revenue augmented through this property.
Partition suit and separate possession of her share shall be the last option if all other efforts including an amicable talks or family arrangement fails to fetch any fruitful result.